pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: Fix visibility check when XID is committed in CLOG but not in pr
6+ messages / 1 participants
[nested] [flat]
* pgsql: Fix visibility check when XID is committed in CLOG but not in pr
@ 2022-06-27 05:25 Heikki Linnakangas <heikki.linnakangas@iki.fi>
0 siblings, 0 replies; 6+ messages in thread
From: Heikki Linnakangas @ 2022-06-27 05:25 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix visibility check when XID is committed in CLOG but not in procarray.
TransactionIdIsInProgress had a fast path to return 'false' if the
single-item CLOG cache said that the transaction was known to be
committed. However, that was wrong, because a transaction is first
marked as committed in the CLOG but doesn't become visible to others
until it has removed its XID from the proc array. That could lead to an
error:
ERROR: t_xmin is uncommitted in tuple to be updated
or for an UPDATE to go ahead without blocking, before the previous
UPDATE on the same row was made visible.
The window is usually very short, but synchronous replication makes it
much wider, because the wait for synchronous replica happens in that
window.
Another thing that makes it hard to hit is that it's hard to get such
a commit-in-progress transaction into the single item CLOG cache.
Normally, if you call TransactionIdIsInProgress on such a transaction,
it determines that the XID is in progress without checking the CLOG
and without populating the cache. One way to prime the cache is to
explicitly call pg_xact_status() on the XID. Another way is to use a
lot of subtransactions, so that the subxid cache in the proc array is
overflown, making TransactionIdIsInProgress rely on pg_subtrans and
CLOG checks.
This has been broken ever since it was introduced in 2008, but the race
condition is very hard to hit, especially without synchronous
replication. There were a couple of reports of the error starting from
summer 2021, but no one was able to find the root cause then.
TransactionIdIsKnownCompleted() is now unused. In 'master', remove it,
but I left it in place in backbranches in case it's used by extensions.
Also change pg_xact_status() to check TransactionIdIsInProgress().
Previously, it only checked the CLOG, and returned "committed" before
the transaction was actually made visible to other queries. Note that
this also means that you cannot use pg_xact_status() to reproduce the
bug anymore, even if the code wasn't fixed.
Report and analysis by Konstantin Knizhnik. Patch by Simon Riggs, with
the pg_xact_status() change added by me.
Author: Simon Riggs
Reviewed-by: Andres Freund
Discussion: https://www.postgresql.org/message-id/flat/4da7913d-398c-e2ad-d777-f752cf7f0bbb%40garret.ru
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/adf6d5dfb2094cc2bb28b4dea0565dcd0e775ed9
Modified Files
--------------
src/backend/access/transam/transam.c | 27 ---------------------------
src/backend/storage/ipc/procarray.c | 12 +++++++++++-
src/backend/utils/adt/xid8funcs.c | 30 ++++++++++++------------------
src/include/access/transam.h | 1 -
4 files changed, 23 insertions(+), 47 deletions(-)
^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix visibility check when XID is committed in CLOG but not in pr
@ 2022-06-27 05:25 Heikki Linnakangas <heikki.linnakangas@iki.fi>
0 siblings, 0 replies; 6+ messages in thread
From: Heikki Linnakangas @ 2022-06-27 05:25 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix visibility check when XID is committed in CLOG but not in procarray.
TransactionIdIsInProgress had a fast path to return 'false' if the
single-item CLOG cache said that the transaction was known to be
committed. However, that was wrong, because a transaction is first
marked as committed in the CLOG but doesn't become visible to others
until it has removed its XID from the proc array. That could lead to an
error:
ERROR: t_xmin is uncommitted in tuple to be updated
or for an UPDATE to go ahead without blocking, before the previous
UPDATE on the same row was made visible.
The window is usually very short, but synchronous replication makes it
much wider, because the wait for synchronous replica happens in that
window.
Another thing that makes it hard to hit is that it's hard to get such
a commit-in-progress transaction into the single item CLOG cache.
Normally, if you call TransactionIdIsInProgress on such a transaction,
it determines that the XID is in progress without checking the CLOG
and without populating the cache. One way to prime the cache is to
explicitly call pg_xact_status() on the XID. Another way is to use a
lot of subtransactions, so that the subxid cache in the proc array is
overflown, making TransactionIdIsInProgress rely on pg_subtrans and
CLOG checks.
This has been broken ever since it was introduced in 2008, but the race
condition is very hard to hit, especially without synchronous
replication. There were a couple of reports of the error starting from
summer 2021, but no one was able to find the root cause then.
TransactionIdIsKnownCompleted() is now unused. In 'master', remove it,
but I left it in place in backbranches in case it's used by extensions.
Also change pg_xact_status() to check TransactionIdIsInProgress().
Previously, it only checked the CLOG, and returned "committed" before
the transaction was actually made visible to other queries. Note that
this also means that you cannot use pg_xact_status() to reproduce the
bug anymore, even if the code wasn't fixed.
Report and analysis by Konstantin Knizhnik. Patch by Simon Riggs, with
the pg_xact_status() change added by me.
Author: Simon Riggs
Reviewed-by: Andres Freund
Discussion: https://www.postgresql.org/message-id/flat/4da7913d-398c-e2ad-d777-f752cf7f0bbb%40garret.ru
Branch
------
REL_14_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/e24615a0057a9932904317576cf5c4d42349b363
Modified Files
--------------
src/backend/access/transam/transam.c | 13 +++++++++----
src/backend/storage/ipc/procarray.c | 12 +++++++++++-
src/backend/utils/adt/xid8funcs.c | 30 ++++++++++++------------------
3 files changed, 32 insertions(+), 23 deletions(-)
^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix visibility check when XID is committed in CLOG but not in pr
@ 2022-06-27 05:25 Heikki Linnakangas <heikki.linnakangas@iki.fi>
0 siblings, 0 replies; 6+ messages in thread
From: Heikki Linnakangas @ 2022-06-27 05:25 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix visibility check when XID is committed in CLOG but not in procarray.
TransactionIdIsInProgress had a fast path to return 'false' if the
single-item CLOG cache said that the transaction was known to be
committed. However, that was wrong, because a transaction is first
marked as committed in the CLOG but doesn't become visible to others
until it has removed its XID from the proc array. That could lead to an
error:
ERROR: t_xmin is uncommitted in tuple to be updated
or for an UPDATE to go ahead without blocking, before the previous
UPDATE on the same row was made visible.
The window is usually very short, but synchronous replication makes it
much wider, because the wait for synchronous replica happens in that
window.
Another thing that makes it hard to hit is that it's hard to get such
a commit-in-progress transaction into the single item CLOG cache.
Normally, if you call TransactionIdIsInProgress on such a transaction,
it determines that the XID is in progress without checking the CLOG
and without populating the cache. One way to prime the cache is to
explicitly call pg_xact_status() on the XID. Another way is to use a
lot of subtransactions, so that the subxid cache in the proc array is
overflown, making TransactionIdIsInProgress rely on pg_subtrans and
CLOG checks.
This has been broken ever since it was introduced in 2008, but the race
condition is very hard to hit, especially without synchronous
replication. There were a couple of reports of the error starting from
summer 2021, but no one was able to find the root cause then.
TransactionIdIsKnownCompleted() is now unused. In 'master', remove it,
but I left it in place in backbranches in case it's used by extensions.
Also change pg_xact_status() to check TransactionIdIsInProgress().
Previously, it only checked the CLOG, and returned "committed" before
the transaction was actually made visible to other queries. Note that
this also means that you cannot use pg_xact_status() to reproduce the
bug anymore, even if the code wasn't fixed.
Report and analysis by Konstantin Knizhnik. Patch by Simon Riggs, with
the pg_xact_status() change added by me.
Author: Simon Riggs
Reviewed-by: Andres Freund
Discussion: https://www.postgresql.org/message-id/flat/4da7913d-398c-e2ad-d777-f752cf7f0bbb%40garret.ru
Branch
------
REL_13_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/7ba325fd7fd450688afd2f80d19cad060feb51b5
Modified Files
--------------
src/backend/access/transam/transam.c | 13 +++++++++----
src/backend/storage/ipc/procarray.c | 12 +++++++++++-
src/backend/utils/adt/xid8funcs.c | 30 ++++++++++++------------------
3 files changed, 32 insertions(+), 23 deletions(-)
^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix visibility check when XID is committed in CLOG but not in pr
@ 2022-06-27 05:25 Heikki Linnakangas <heikki.linnakangas@iki.fi>
0 siblings, 0 replies; 6+ messages in thread
From: Heikki Linnakangas @ 2022-06-27 05:25 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix visibility check when XID is committed in CLOG but not in procarray.
TransactionIdIsInProgress had a fast path to return 'false' if the
single-item CLOG cache said that the transaction was known to be
committed. However, that was wrong, because a transaction is first
marked as committed in the CLOG but doesn't become visible to others
until it has removed its XID from the proc array. That could lead to an
error:
ERROR: t_xmin is uncommitted in tuple to be updated
or for an UPDATE to go ahead without blocking, before the previous
UPDATE on the same row was made visible.
The window is usually very short, but synchronous replication makes it
much wider, because the wait for synchronous replica happens in that
window.
Another thing that makes it hard to hit is that it's hard to get such
a commit-in-progress transaction into the single item CLOG cache.
Normally, if you call TransactionIdIsInProgress on such a transaction,
it determines that the XID is in progress without checking the CLOG
and without populating the cache. One way to prime the cache is to
explicitly call pg_xact_status() on the XID. Another way is to use a
lot of subtransactions, so that the subxid cache in the proc array is
overflown, making TransactionIdIsInProgress rely on pg_subtrans and
CLOG checks.
This has been broken ever since it was introduced in 2008, but the race
condition is very hard to hit, especially without synchronous
replication. There were a couple of reports of the error starting from
summer 2021, but no one was able to find the root cause then.
TransactionIdIsKnownCompleted() is now unused. In 'master', remove it,
but I left it in place in backbranches in case it's used by extensions.
Also change pg_xact_status() to check TransactionIdIsInProgress().
Previously, it only checked the CLOG, and returned "committed" before
the transaction was actually made visible to other queries. Note that
this also means that you cannot use pg_xact_status() to reproduce the
bug anymore, even if the code wasn't fixed.
Report and analysis by Konstantin Knizhnik. Patch by Simon Riggs, with
the pg_xact_status() change added by me.
Author: Simon Riggs
Reviewed-by: Andres Freund
Discussion: https://www.postgresql.org/message-id/flat/4da7913d-398c-e2ad-d777-f752cf7f0bbb%40garret.ru
Branch
------
REL_12_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/af530898ebc813941a6ec82856b0e06c12facc52
Modified Files
--------------
src/backend/access/transam/transam.c | 13 +++++++++----
src/backend/storage/ipc/procarray.c | 12 +++++++++++-
src/backend/utils/adt/txid.c | 30 ++++++++++++------------------
3 files changed, 32 insertions(+), 23 deletions(-)
^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix visibility check when XID is committed in CLOG but not in pr
@ 2022-06-27 05:25 Heikki Linnakangas <heikki.linnakangas@iki.fi>
0 siblings, 0 replies; 6+ messages in thread
From: Heikki Linnakangas @ 2022-06-27 05:25 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix visibility check when XID is committed in CLOG but not in procarray.
TransactionIdIsInProgress had a fast path to return 'false' if the
single-item CLOG cache said that the transaction was known to be
committed. However, that was wrong, because a transaction is first
marked as committed in the CLOG but doesn't become visible to others
until it has removed its XID from the proc array. That could lead to an
error:
ERROR: t_xmin is uncommitted in tuple to be updated
or for an UPDATE to go ahead without blocking, before the previous
UPDATE on the same row was made visible.
The window is usually very short, but synchronous replication makes it
much wider, because the wait for synchronous replica happens in that
window.
Another thing that makes it hard to hit is that it's hard to get such
a commit-in-progress transaction into the single item CLOG cache.
Normally, if you call TransactionIdIsInProgress on such a transaction,
it determines that the XID is in progress without checking the CLOG
and without populating the cache. One way to prime the cache is to
explicitly call pg_xact_status() on the XID. Another way is to use a
lot of subtransactions, so that the subxid cache in the proc array is
overflown, making TransactionIdIsInProgress rely on pg_subtrans and
CLOG checks.
This has been broken ever since it was introduced in 2008, but the race
condition is very hard to hit, especially without synchronous
replication. There were a couple of reports of the error starting from
summer 2021, but no one was able to find the root cause then.
TransactionIdIsKnownCompleted() is now unused. In 'master', remove it,
but I left it in place in backbranches in case it's used by extensions.
Also change pg_xact_status() to check TransactionIdIsInProgress().
Previously, it only checked the CLOG, and returned "committed" before
the transaction was actually made visible to other queries. Note that
this also means that you cannot use pg_xact_status() to reproduce the
bug anymore, even if the code wasn't fixed.
Report and analysis by Konstantin Knizhnik. Patch by Simon Riggs, with
the pg_xact_status() change added by me.
Author: Simon Riggs
Reviewed-by: Andres Freund
Discussion: https://www.postgresql.org/message-id/flat/4da7913d-398c-e2ad-d777-f752cf7f0bbb%40garret.ru
Branch
------
REL_11_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/b49889f3c0fa3e6bf5ad1df813758767aa51bc31
Modified Files
--------------
src/backend/access/transam/transam.c | 13 +++++++++----
src/backend/storage/ipc/procarray.c | 12 +++++++++++-
src/backend/utils/adt/txid.c | 30 ++++++++++++------------------
3 files changed, 32 insertions(+), 23 deletions(-)
^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fix visibility check when XID is committed in CLOG but not in pr
@ 2022-06-27 05:25 Heikki Linnakangas <heikki.linnakangas@iki.fi>
0 siblings, 0 replies; 6+ messages in thread
From: Heikki Linnakangas @ 2022-06-27 05:25 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix visibility check when XID is committed in CLOG but not in procarray.
TransactionIdIsInProgress had a fast path to return 'false' if the
single-item CLOG cache said that the transaction was known to be
committed. However, that was wrong, because a transaction is first
marked as committed in the CLOG but doesn't become visible to others
until it has removed its XID from the proc array. That could lead to an
error:
ERROR: t_xmin is uncommitted in tuple to be updated
or for an UPDATE to go ahead without blocking, before the previous
UPDATE on the same row was made visible.
The window is usually very short, but synchronous replication makes it
much wider, because the wait for synchronous replica happens in that
window.
Another thing that makes it hard to hit is that it's hard to get such
a commit-in-progress transaction into the single item CLOG cache.
Normally, if you call TransactionIdIsInProgress on such a transaction,
it determines that the XID is in progress without checking the CLOG
and without populating the cache. One way to prime the cache is to
explicitly call pg_xact_status() on the XID. Another way is to use a
lot of subtransactions, so that the subxid cache in the proc array is
overflown, making TransactionIdIsInProgress rely on pg_subtrans and
CLOG checks.
This has been broken ever since it was introduced in 2008, but the race
condition is very hard to hit, especially without synchronous
replication. There were a couple of reports of the error starting from
summer 2021, but no one was able to find the root cause then.
TransactionIdIsKnownCompleted() is now unused. In 'master', remove it,
but I left it in place in backbranches in case it's used by extensions.
Also change pg_xact_status() to check TransactionIdIsInProgress().
Previously, it only checked the CLOG, and returned "committed" before
the transaction was actually made visible to other queries. Note that
this also means that you cannot use pg_xact_status() to reproduce the
bug anymore, even if the code wasn't fixed.
Report and analysis by Konstantin Knizhnik. Patch by Simon Riggs, with
the pg_xact_status() change added by me.
Author: Simon Riggs
Reviewed-by: Andres Freund
Discussion: https://www.postgresql.org/message-id/flat/4da7913d-398c-e2ad-d777-f752cf7f0bbb%40garret.ru
Branch
------
REL_10_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/4822b462726188fa6aa79c41bdb30ae54e84c596
Modified Files
--------------
src/backend/access/transam/transam.c | 13 +++++++++----
src/backend/storage/ipc/procarray.c | 12 +++++++++++-
src/backend/utils/adt/txid.c | 30 ++++++++++++------------------
3 files changed, 32 insertions(+), 23 deletions(-)
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2022-06-27 05:25 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-06-27 05:25 pgsql: Fix visibility check when XID is committed in CLOG but not in pr Heikki Linnakangas <heikki.linnakangas@iki.fi>
2022-06-27 05:25 pgsql: Fix visibility check when XID is committed in CLOG but not in pr Heikki Linnakangas <heikki.linnakangas@iki.fi>
2022-06-27 05:25 pgsql: Fix visibility check when XID is committed in CLOG but not in pr Heikki Linnakangas <heikki.linnakangas@iki.fi>
2022-06-27 05:25 pgsql: Fix visibility check when XID is committed in CLOG but not in pr Heikki Linnakangas <heikki.linnakangas@iki.fi>
2022-06-27 05:25 pgsql: Fix visibility check when XID is committed in CLOG but not in pr Heikki Linnakangas <heikki.linnakangas@iki.fi>
2022-06-27 05:25 pgsql: Fix visibility check when XID is committed in CLOG but not in pr Heikki Linnakangas <heikki.linnakangas@iki.fi>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox