agora inbox for pgsql-hackers@postgresql.orghelp / color / mirror / Atom feed
[PATCH] Fix race conditions during the setup of logical decoding. 525+ messages / 2 participants [nested] [flat]
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 41 +++++++++++++++++++ .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 42 insertions(+) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 9b09dc8eac1..c02d08f3417 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -400,6 +400,47 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + while (true) + { + bool found = false; + + for (int i = 0; i < builder->committed.xcnt; i++) + { + /* + * XXX Is it worth remembering the XIDs that appear to be + * committed per CLOG and skipping them in the next iteration of + * the outer loop? Not sure it's worth the effort - a single + * iteration is enough in most cases. + */ + if (unlikely(!TransactionIdDidCommit(builder->committed.xip[i]))) + { + found = true; + + /* + * Wait a bit before going to the next iteration of the outer + * loop. The race conditions we address here is pretty rare, + * so we shouldn't need to wait too long. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + + break; + } + } + + if (!found) + break; + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..b6318b0cf37 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH 1/2] Fix race conditions during the setup of logical decoding. @ 2026-01-19 15:07 Antonin Houska <ah@cybertec.at> 0 siblings, 0 replies; 525+ messages in thread From: Antonin Houska @ 2026-01-19 15:07 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. --- src/backend/replication/logical/snapbuild.c | 27 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index 37f0c6028bd..d7ea098cb37 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -377,7 +377,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -402,6 +402,31 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although it's very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (TransactionIdDidCommit(builder->committed.xip[i])) + break; + else + { + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 10L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 4aa864fe3c3..987df777e47 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -181,6 +181,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --obdbyrpgpb3edptv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0002-What-about-just-ignoring-the-xacts-if-they-re-still-.patch" ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
* [PATCH v3] Fix race conditions during the setup of logical decoding. @ 2026-08-21 17:52 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 525+ messages in thread From: Álvaro Herrera @ 2026-08-21 17:52 UTC (permalink / raw) Although it's rather unlikely, it can happen that the snapshot builder considers transaction committed (according to WAL) before the commit could be recorded in CLOG. In an extreme case, snapshot can even be created and used in between. Since both snapshot and CLOG are needed for visibility checks, this inconsistency can make them work incorrectly. The typical symptom is that a transaction that the snapshot considers not running anymore is (per CLOG) considered aborted instead of committed. Thus a new tuple version can be evaluated as invisible (if xmin is incorrectly considered aborted) or a deleted tuple version can be evaluated as visible (if xmax is incorrectly considered aborted). This patch fixes the problem by checking if all the XIDs that the new snapshot considers committed are really committed per CLOG. If at least one is not, the check is repeated after a short delay. However, a single check is sufficient in almost all cases, so the performance impact should be minimal. Author: Antonin Houska <ah@cybertec.at> Discussion: https://postgr.es/m/85833.1768840165@localhost --- src/backend/replication/logical/snapbuild.c | 74 ++++++++++++++++++- .../utils/activity/wait_event_names.txt | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index f60bcf09605..31608c186dc 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -130,6 +130,7 @@ #include "access/xact.h" #include "common/file_utils.h" #include "miscadmin.h" +#include "nodes/pg_list.h" #include "pgstat.h" #include "replication/logical.h" #include "replication/reorderbuffer.h" @@ -365,6 +366,8 @@ SnapBuildBuildSnapshot(SnapBuild *builder) { Snapshot snapshot; Size ssize; + static List *xids_already_tested = NIL; + static List *new_xids_already_tested = NIL; Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT); @@ -378,7 +381,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) /* * We misuse the original meaning of SnapshotData's xip and subxip fields - * to make the more fitting for our needs. + * to make them more fitting for our needs. * * In the 'xip' array we store transactions that have to be treated as * committed. Since we will only ever look at tuples from transactions @@ -403,6 +406,75 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->xmin = builder->xmin; snapshot->xmax = builder->xmax; + /* + * Although very unlikely, it's possible that a commit WAL record was + * decoded but CLOG is not aware of the commit yet. Should the CLOG update + * be delayed even more, visibility checks that use this snapshot could + * work incorrectly. Therefore we check the CLOG status here. + * + * We must not do this using TransactionIdIsInProgress()! The check there + * for latestCompletedXid would wreak havoc because the transaction we're + * interested in may not be out of ProcArray yet, but we must not wait for + * that. Doing the transam.c check directly is correct, though unusual. + * + * We don't want to repeatedly read the CLOG status for the same + * transaction, and it's easy to keep track of which ones we've already + * checked. Keep a list of the ones we test on each cycle, and use the + * list from the previous cycle to skip testing them now. + */ + for (int i = 0; i < builder->committed.xcnt; i++) + { + for (;;) + { + if (list_member_xid(xids_already_tested, builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else if (TransactionIdDidCommit(builder->committed.xip[i])) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + new_xids_already_tested = lappend_xid(new_xids_already_tested, + builder->committed.xip[i]); + MemoryContextSwitchTo(oldcxt); + break; + } + else + { + /* + * Note that the other process doesn't know we're waiting for + * them, so nothing is going to signal us out of this latch. + * Therefore use a short timeout. + */ + (void) WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | + WL_EXIT_ON_PM_DEATH, + 2L, + WAIT_EVENT_SNAPBUILD_CLOG); + ResetLatch(MyLatch); + } + CHECK_FOR_INTERRUPTS(); + } + } + + /* Swap these lists for next time */ + if (xids_already_tested != NIL) + list_free(xids_already_tested); + if (new_xids_already_tested != NIL) + { + xids_already_tested = new_xids_already_tested; + new_xids_already_tested = NIL; + } + else + xids_already_tested = NIL; + /* store all transactions to be treated as committed by this snapshot */ snapshot->xip = (TransactionId *) ((char *) snapshot + sizeof(SnapshotData)); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 256b3a3c02e..55a9c8296b5 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -185,6 +185,7 @@ PG_SLEEP "Waiting due to a call to <function>pg_sleep</function> or a sibling fu RECOVERY_APPLY_DELAY "Waiting to apply WAL during recovery because of a delay setting." RECOVERY_RETRIEVE_RETRY_INTERVAL "Waiting during recovery when WAL data is not available from any source (<filename>pg_wal</filename>, archive or stream)." REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the checkpointer, because the request queue is full." +SNAPBUILD_CLOG "Waiting for CLOG update before building snapshot." SPIN_DELAY "Waiting while acquiring a contended spinlock." VACUUM_DELAY "Waiting in a cost-based vacuum delay point." VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed." -- 2.47.3 --ic4mjwhsvm7ysii6-- ^ permalink raw reply [nested|flat] 525+ messages in thread
end of thread, other threads:[~2026-08-21 17:52 UTC | newest] Thread overview: 525+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH 1/2] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-01-19 15:07 [PATCH] Fix race conditions during the setup of logical decoding. Antonin Houska <ah@cybertec.at> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de> 2026-08-21 17:52 [PATCH v3] Fix race conditions during the setup of logical decoding. Álvaro Herrera <alvherre@kurilemu.de>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox