pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
9+ messages / 4 participants
[nested] [flat]
Commits from this thread:
2021-06-28 741deb2603 c62c3769ff b75c1f6879 b786304c29 Fix race condition in TransactionGroupUpdateXidStatus().
* BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-24 16:00 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 3 replies; 9+ messages in thread
From: PG Bug reporting form @ 2021-06-24 16:00 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: exclusion@gmail.com
The following bug has been logged on the website:
Bug reference: 17072
Logged by: Alexander Lakhin
Email address: exclusion@gmail.com
PostgreSQL version: 13.3
Operating system: Ubuntu 20.04
Description:
When running multiple installchecks (100-200) in a loop I get the following
fail:
Core was generated by `postgres: postgres regress119 127.0.0.1(60410) in'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f1c7203d535 in __GI_abort () at abort.c:79
#2 0x00005645576b41a2 in ExceptionalCondition (
conditionName=conditionName@entry=0x564557840fc8
"pg_atomic_read_u32(&MyProc->clogGroupNext) == INVALID_PGPROCNO",
errorType=errorType@entry=0x56455771001d "FailedAssertion",
fileName=fileName@entry=0x56455774676d "proc.c",
lineNumber=lineNumber@entry=498) at assert.c:67
#3 0x000056455757b062 in InitProcess () at proc.c:498
#4 0x00005645575893df in PostgresMain (argc=1,
argv=argv@entry=0x5645581d5820, dbname=<optimized out>,
username=0x5645581d5168 "postgres") at postgres.c:4393
#5 0x00005645574e6f91 in BackendRun (port=0x5645581c73d0,
port=0x5645581c73d0) at postmaster.c:5784
#6 BackendStartup (pool=pool@entry=0x0, port=port@entry=0x5645581c73d0) at
postmaster.c:5440
#7 0x00005645574e769b in ServerLoop () at postmaster.c:2403
#8 0x00005645574e8e4b in PostmasterMain (argc=<optimized out>,
argv=<optimized out>) at postmaster.c:1870
#9 0x00005645571db177 in main (argc=3, argv=0x56455819ad30) at main.c:211
(gdb) print MyProc->clogGroupNext
$1 = {value = 985}
(gdb) print &MyProc->clogGroupNext
$2 = (pg_atomic_uint32 *) 0x7f1c64806a0c
With some diagnostic output added I've got in the server log:
2021-06-24 16:44:16.018 MSK|postgres|regress184|60d48c2d.ea21|WARNING:
TransactionGroupUpdateXidStatus: 819, 2147483647
...
2021-06-24 16:44:16.018 MSK|postgres|regress54|60d48c27.df96|WARNING:
TransactionGroupUpdateXidStatus: 982, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress33|60d48c2f.efdc|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648dd8ec, 974
2021-06-24 16:44:16.018 MSK|postgres|regress44|60d48c2a.e5a7|WARNING:
TransactionGroupUpdateXidStatus: 981, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress72|60d48c2d.ea64|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c64880824, 974
2021-06-24 16:44:16.021 MSK|postgres|regress32|60d48c2f.ee50|WARNING:
TransactionGroupUpdateXidStatus: after while: 974
2021-06-24 16:44:16.018 MSK|postgres|regress144|60d48c2e.ebc5|WARNING:
TransactionGroupUpdateXidStatus: 963, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress184|60d48c2d.ea21|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c64806a0c, 2147483647
2021-06-24 16:44:16.018 MSK|postgres|regress149|60d48c2a.e674|WARNING:
TransactionGroupUpdateXidStatus: 799, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress194|60d48c2f.eed2|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648610e4, 944
2021-06-24 16:44:16.018 MSK|postgres|regress114|60d48c29.e3e1|WARNING:
TransactionGroupUpdateXidStatus: 816, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress33|60d48c2f.efdc|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648dd8ec, 944
2021-06-24 16:44:16.021 MSK|postgres|regress186|60d48c2f.eea9|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648e95a4, 2147483647
2021-06-24 16:44:16.018 MSK|postgres|regress147|60d48c2f.efb7|ERROR: cannot
change routine kind
2021-06-24 16:44:16.018 MSK|postgres|regress147|60d48c2f.efb7|DETAIL:
"functest1" is a function.
2021-06-24 16:44:16.021 MSK|postgres|regress72|60d48c2d.ea64|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c64880824, 944
2021-06-24 16:44:16.021 MSK|postgres|regress50|60d48c2b.e710|ERROR:
relation "does_not_exist" does not exist
2021-06-24 16:44:16.020 MSK|postgres|regress22|60d48c2f.efed|WARNING:
TransactionGroupUpdateXidStatus: after while: 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress54|60d48c27.df96|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648dc3f4, 2147483647
2021-06-24 16:44:16.020 MSK|postgres|regress36|60d48c2f.eff6|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 INVALID_PGPROCNO:
0x7f1c64828b3c, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress34|60d48c2f.ee8b|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648e02dc, 2147483647
2021-06-24 16:44:16.020 MSK|postgres|regress11|60d48c2f.ef8e|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c647f6e6c, 792
2021-06-24 16:44:16.021 MSK|postgres|regress44|60d48c2a.e5a7|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648daefc, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress184|60d48c2d.ea21|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c64806a0c, 944
2021-06-24 16:44:16.021 MSK|postgres|regress194|60d48c2f.eed2|WARNING:
TransactionGroupUpdateXidStatus: after while: 944
2021-06-24 16:44:16.020 MSK|postgres|regress155|60d48c2f.efe1|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648ad114, 792
2021-06-24 16:44:16.021 MSK|postgres|regress33|60d48c2f.efdc|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648dd8ec, 888
2021-06-24 16:44:16.021 MSK|postgres|regress186|60d48c2f.eea9|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648e95a4, 888
2021-06-24 16:44:16.021 MSK|postgres|regress72|60d48c2d.ea64|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c64880824, 888
2021-06-24 16:44:16.021 MSK|postgres|regress54|60d48c27.df96|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648dc3f4, 888
2021-06-24 16:44:16.021 MSK|postgres|regress144|60d48c2e.ebc5|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648c358c, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress186|60d48c2f.eea9|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648e95a4, 985
2021-06-24 16:44:16.021 MSK|postgres|regress34|60d48c2f.ee8b|WARNING:
TransactionGroupUpdateXidStatus: after while: 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress149|60d48c2a.e674|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c647ec6ac, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress44|60d48c2a.e5a7|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648daefc, 985
2021-06-24 16:44:16.021 MSK|postgres|regress22|60d48c2f.efed|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 INVALID_PGPROCNO:
0x7f1c648610e4, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress36|60d48c2f.eff6|WARNING:
TransactionGroupUpdateXidStatus: end
2021-06-24 16:44:16.021 MSK|postgres|regress184|60d48c2d.ea21|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c64806a0c, 985
2021-06-24 16:44:16.021 MSK|postgres|regress114|60d48c29.e3e1|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c64802b24, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress11|60d48c2f.ef8e|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c647f6e6c, 985
2021-06-24 16:44:16.021 MSK|postgres|regress33|60d48c2f.efdc|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648dd8ec, 985
2021-06-24 16:44:16.021 MSK|postgres|regress50|60d48c2b.e710|ERROR: syntax
error at or near "arg" at character 22
2021-06-24 16:44:16.021 MSK|postgres|regress155|60d48c2f.efe1|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648ad114, 985
2021-06-24 16:44:16.021 MSK|postgres|regress72|60d48c2d.ea64|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c64880824, 985
2021-06-24 16:44:16.021 MSK|postgres|regress54|60d48c27.df96|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx:
0x7f1c648dc3f4, 985
2021-06-24 16:44:16.021 MSK|postgres|regress186|60d48c2f.eea9|WARNING:
TransactionGroupUpdateXidStatus: after while: 985
2021-06-24 16:44:16.021 MSK|postgres|regress34|60d48c2f.ee8b|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 INVALID_PGPROCNO:
0x7f1c648e95a4, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress144|60d48c2e.ebc5|WARNING:
TransactionGroupUpdateXidStatus: after while: 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress44|60d48c2a.e5a7|WARNING:
TransactionGroupUpdateXidStatus: return false (963)
2021-06-24 16:44:16.021 MSK|postgres|regress22|60d48c2f.efed|WARNING:
TransactionGroupUpdateXidStatus: pg_atomic_write_u32 INVALID_PGPROCNO:
0x7f1c648aa724, 2147483647
2021-06-24 16:44:16.021 MSK|postgres|regress184|60d48c2d.ea21|WARNING:
TransactionGroupUpdateXidStatus: return false (963)
2021-06-24 16:44:16.021 MSK|postgres|regress194|60d48c2f.eed2|WARNING:
TransactionGroupUpdateXidStatus: return true (944)
2021-06-24 16:44:16.021 MSK|postgres|regress11|60d48c2f.ef8e|WARNING:
TransactionGroupUpdateXidStatus: return false (963)
2021-06-24 16:44:16.021 MSK|postgres|regress33|60d48c2f.efdc|WARNING:
TransactionGroupUpdateXidStatus: return false (963)
The offending (the one that leaved a "valid" clogGroupNext) proccess is
60d48c2d.ea21. It looks like it got from the
pg_atomic_compare_exchange_u32() the nextidx value that was written in the
clogGroupFirst by the process 60d48c2e.ebc5, and exited just after that.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-24 16:15 Alexander Lakhin <exclusion@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
2 siblings, 1 reply; 9+ messages in thread
From: Alexander Lakhin @ 2021-06-24 16:15 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
24.06.2021 19:00, PG Bug reporting form wrote:
> With some diagnostic output added I've got in the server log:
The diagnostic patch is attached.
I could not find an easy reproduction for the issue, so I can propose
only dirty one, with the special debug patch and the following script:
for i in `seq 100`; do
createdb db$i
done
echo "
CREATE TABLE tst (i INT);
INSERT INTO tst SELECT i FROM generate_series(1,100) i;
DROP TABLE tst;
" >/tmp/action.sql
for n in `seq 100`; do
echo "iteration $n"
for i in `seq 100`; do
( { for f in `seq 10`; do cat /tmp/action.sql; done } | psql -d db$i
) >psql-$i.log 2>&1 &
done
wait
coredumpctl --no-pager && break
done
(You need to set max_connections = 100 in postgresql.conf.)
This script with the patched server fails for me on iterations 66, 12, 12.
Best regards,
Alexander
Attachments: download all (2 files, .tar.gz)
clog-diagnostics.patch (text/x-patch, 2.4K) download
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index 880f18c1473..126502363f5 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -432,7 +432,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
proc->clogGroupMemberLsn = lsn;
nextidx = pg_atomic_read_u32(&procglobal->clogGroupFirst);
-
+elog(WARNING, "TransactionGroupUpdateXidStatus: %d, %d", proc->pgprocno, nextidx);
while (true)
{
/*
@@ -451,9 +451,10 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
ProcGlobal->allProcs[nextidx].clogGroupMemberPage != proc->clogGroupMemberPage)
{
proc->clogGroupMember = false;
+elog(WARNING, "TransactionGroupUpdateXidStatus: return false (%d)", nextidx);
return false;
}
-
+elog(WARNING, "TransactionGroupUpdateXidStatus: pg_atomic_write_u32 nextidx: %p, %d", &proc->clogGroupNext, nextidx);
pg_atomic_write_u32(&proc->clogGroupNext, nextidx);
if (pg_atomic_compare_exchange_u32(&procglobal->clogGroupFirst,
@@ -461,7 +462,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
(uint32) proc->pgprocno))
break;
}
-
+elog(WARNING, "TransactionGroupUpdateXidStatus: after while: %d", nextidx);
/*
* If the list was not empty, the leader will update the status of our
* XID. It is impossible to have followers without a leader because the
@@ -489,6 +490,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
/* Fix semaphore count for any absorbed wakeups */
while (extraWaits-- > 0)
PGSemaphoreUnlock(proc->sem);
+elog(WARNING, "TransactionGroupUpdateXidStatus: return true (%d)", nextidx);
return true;
}
@@ -540,8 +542,8 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
while (wakeidx != INVALID_PGPROCNO)
{
PGPROC *proc = &ProcGlobal->allProcs[wakeidx];
-
wakeidx = pg_atomic_read_u32(&proc->clogGroupNext);
+elog(WARNING, "TransactionGroupUpdateXidStatus: pg_atomic_write_u32 INVALID_PGPROCNO: %p, %d", &proc->clogGroupNext, INVALID_PGPROCNO);
pg_atomic_write_u32(&proc->clogGroupNext, INVALID_PGPROCNO);
/* ensure all previous writes are visible before follower continues. */
@@ -553,6 +555,7 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
PGSemaphoreUnlock(proc->sem);
}
+elog(WARNING, "TransactionGroupUpdateXidStatus: end");
return true;
}
clog-debug.patch (text/x-patch, 1.4K) download
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index 880f18c1473..499989cb3a1 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -305,7 +305,7 @@ TransactionIdSetPageStatus(TransactionId xid, int nsubxids,
* update. If that doesn't work out, fall back to waiting for the
* lock to perform an update for this transaction only.
*/
- if (LWLockConditionalAcquire(XactSLRULock, LW_EXCLUSIVE))
+ if (0)
{
/* Got the lock without waiting! Do the update. */
TransactionIdSetPageStatusInternal(xid, nsubxids, subxids, status,
@@ -432,7 +432,6 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
proc->clogGroupMemberLsn = lsn;
nextidx = pg_atomic_read_u32(&procglobal->clogGroupFirst);
-
while (true)
{
/*
@@ -451,10 +450,15 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
ProcGlobal->allProcs[nextidx].clogGroupMemberPage != proc->clogGroupMemberPage)
{
proc->clogGroupMember = false;
+elog(WARNING, "TransactionGroupUpdateXidStatus: return false");
return false;
}
-
pg_atomic_write_u32(&proc->clogGroupNext, nextidx);
+do {
+instr_time t;
+INSTR_TIME_SET_CURRENT(t);
+if ((t.tv_nsec % 1000000L) < 10L) break;
+} while (1);
if (pg_atomic_compare_exchange_u32(&procglobal->clogGroupFirst,
&nextidx,
* Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-25 09:55 Amit Kapila <amit.kapila16@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
2 siblings, 1 reply; 9+ messages in thread
From: Amit Kapila @ 2021-06-25 09:55 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Jun 25, 2021 at 12:20 AM PG Bug reporting form
<noreply@postgresql.org> wrote:
>
> The offending (the one that leaved a "valid" clogGroupNext) proccess is
> 60d48c2d.ea21. It looks like it got from the
> pg_atomic_compare_exchange_u32() the nextidx value that was written in the
> clogGroupFirst by the process 60d48c2e.ebc5, and exited just after that.
>
Your analysis seems to be in the right direction. Can you try by
setting clogGroupNext to INVALID_PGPROCNO
(pg_atomic_write_u32(&proc->clogGroupNext, INVALID_PGPROCNO);) before
we return false in the first while(true) loop in function
TransactionGroupUpdateXidStatus()?
I think this should be reproducible on all branches from HEAD till
v11. Have you tried in any other branch? I'll also try to reproduce
it.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-25 10:00 Alexander Lakhin <exclusion@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
2 siblings, 0 replies; 9+ messages in thread
From: Alexander Lakhin @ 2021-06-25 10:00 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
24.06.2021 19:00, PG Bug reporting form wrote:
> When running multiple installchecks (100-200) in a loop I get the following
> fail:
If it could be of any use, please look at the draft script that executes
multiple installchecks on one instance.
(Sorry for a dirty play with the makefle and neighbors — it's not
intended for the production use.)
I started to develop this script to discover a hardware or software
issue on a powerful server (192 cores, 4 TB RAM, ...).
Single installcheck is a needle in a haystack for such hardware, so I
tried to run multiple instances and multiple installchecks for each.
Some tests are not ready to run in a such configuration, but most of the
tests work and bring some findings.
Best regards,
Alexander
Attachments:
test-icx.sh (application/x-shellscript, 5.5K) download
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-25 10:32 vignesh C <vignesh21@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 0 replies; 9+ messages in thread
From: vignesh C @ 2021-06-25 10:32 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 24, 2021 at 9:45 PM Alexander Lakhin <exclusion@gmail.com> wrote:
>
> 24.06.2021 19:00, PG Bug reporting form wrote:
> > With some diagnostic output added I've got in the server log:
> The diagnostic patch is attached.
>
> I could not find an easy reproduction for the issue, so I can propose
> only dirty one, with the special debug patch and the following script:
>
> for i in `seq 100`; do
> createdb db$i
> done
>
> echo "
> CREATE TABLE tst (i INT);
> INSERT INTO tst SELECT i FROM generate_series(1,100) i;
> DROP TABLE tst;
> " >/tmp/action.sql
>
> for n in `seq 100`; do
> echo "iteration $n"
> for i in `seq 100`; do
> ( { for f in `seq 10`; do cat /tmp/action.sql; done } | psql -d db$i
> ) >psql-$i.log 2>&1 &
> done
> wait
> coredumpctl --no-pager && break
> done
>
> (You need to set max_connections = 100 in postgresql.conf.)
> This script with the patched server fails for me on iterations 66, 12, 12.
I was not able to reproduce the problem with REL_13_3, REL_13_Stable &
Head branch with the script and patch shared. I will check and see if
I can reproduce the problem with the new script that you have shared.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-25 11:00 Alexander Lakhin <exclusion@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Alexander Lakhin @ 2021-06-25 11:00 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hello Amit,
25.06.2021 12:55, Amit Kapila wrote:
> On Fri, Jun 25, 2021 at 12:20 AM PG Bug reporting form
> <noreply@postgresql.org> wrote:
>> The offending (the one that leaved a "valid" clogGroupNext) proccess is
>> 60d48c2d.ea21. It looks like it got from the
>> pg_atomic_compare_exchange_u32() the nextidx value that was written in the
>> clogGroupFirst by the process 60d48c2e.ebc5, and exited just after that.
>>
> Your analysis seems to be in the right direction. Can you try by
> setting clogGroupNext to INVALID_PGPROCNO
> (pg_atomic_write_u32(&proc->clogGroupNext, INVALID_PGPROCNO);) before
> we return false in the first while(true) loop in function
> TransactionGroupUpdateXidStatus()?
With this modification that assert is not triggered, all 100 iterations
pass fine (triple checked).
> I think this should be reproducible on all branches from HEAD till
> v11. Have you tried in any other branch? I'll also try to reproduce
> it.
I've reproduced it on REL_11_STABLE, REL_12_STABLE, REL_13_STABLE, and
master.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-26 09:57 Amit Kapila <amit.kapila16@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Amit Kapila @ 2021-06-26 09:57 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Jun 25, 2021 at 4:30 PM Alexander Lakhin <exclusion@gmail.com> wrote:
>
> Hello Amit,
> 25.06.2021 12:55, Amit Kapila wrote:
> > On Fri, Jun 25, 2021 at 12:20 AM PG Bug reporting form
> > <noreply@postgresql.org> wrote:
> >> The offending (the one that leaved a "valid" clogGroupNext) proccess is
> >> 60d48c2d.ea21. It looks like it got from the
> >> pg_atomic_compare_exchange_u32() the nextidx value that was written in the
> >> clogGroupFirst by the process 60d48c2e.ebc5, and exited just after that.
> >>
> > Your analysis seems to be in the right direction. Can you try by
> > setting clogGroupNext to INVALID_PGPROCNO
> > (pg_atomic_write_u32(&proc->clogGroupNext, INVALID_PGPROCNO);) before
> > we return false in the first while(true) loop in function
> > TransactionGroupUpdateXidStatus()?
> With this modification that assert is not triggered, all 100 iterations
> pass fine (triple checked).
>
Okay, please find the patch for the same attached.
> > I think this should be reproducible on all branches from HEAD till
> > v11. Have you tried in any other branch? I'll also try to reproduce
> > it.
> I've reproduced it on REL_11_STABLE, REL_12_STABLE, REL_13_STABLE, and
> master.
>
Please see if you can verify whether the attached fixes it in all the
branches? I have also reproduced it in a bit different way by using a
debugger. Basically, by having three sessions trying to commit at the
same time. After the first session became the first group member,
allow the second session to check if it can become a member and stop
it via debugger just before it becomes the member. Then, allow the
first session to complete the transaction and allow the third session
to become the group leader (or first group member). After that when
the second session tries to become the member, it will notice that the
leader has changed and again try to become a member of the new leader.
Then, I forced via debugger to allow the second member to return false
and perform the commit by itself. Next, disconnect and connect again
in the second session and we will see assertion failure as reported by
you. The attached patch fixes the assertion failure.
--
With Regards,
Amit Kapila.
Attachments:
v1-0001-Fix-race-condition-in-TransactionGroupUpdateXidSt.patch (application/octet-stream, 1.8K) download
From 4189faa568d7b19931f9be095222f2484c5fc998 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Sat, 26 Jun 2021 15:00:31 +0530
Subject: [PATCH v1] Fix race condition in TransactionGroupUpdateXidStatus().
When we cannot immediately acquire XactSLRULock in exclusive mode at
commit time, we add ourselves to a list of processes that need their XIDs
status update. We do this iff the clog page where we need to update the
current transaction status is the same as the group leader's clog page,
otherwise, we allow the caller to clear it by itself. Now, when we can't
add ourselves to any group, we were not clearing the current proc if it
has already become a member of some group which was leading to an
assertion failure when the same proc was assigned to another backend after
the current backend exits.
Reported-by: Alexander Lakhin
Bug: 17072
Author: Amit Kapila
Tested-By: Alexander Lakhin
Backpatch-through: 11, where it was introduced
Discussion: https://postgr.es/m/17072-2f8764857ef2c92a@postgresql.org
---
src/backend/access/transam/clog.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c
index 6fa4713fb4..3ea16a270a 100644
--- a/src/backend/access/transam/clog.c
+++ b/src/backend/access/transam/clog.c
@@ -451,7 +451,12 @@ TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status,
if (nextidx != INVALID_PGPROCNO &&
ProcGlobal->allProcs[nextidx].clogGroupMemberPage != proc->clogGroupMemberPage)
{
+ /*
+ * Ensure that this proc is not a member of any clog group that
+ * needs an XID status update.
+ */
proc->clogGroupMember = false;
+ pg_atomic_write_u32(&proc->clogGroupNext, INVALID_PGPROCNO);
return false;
}
--
2.28.0.windows.1
* Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-27 19:00 Alexander Lakhin <exclusion@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Alexander Lakhin @ 2021-06-27 19:00 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hello Amit,
26.06.2021 12:57, Amit Kapila wrote:
>
>>> I think this should be reproducible on all branches from HEAD till
>>> v11. Have you tried in any other branch? I'll also try to reproduce
>>> it.
>> I've reproduced it on REL_11_STABLE, REL_12_STABLE, REL_13_STABLE, and
>> master.
>>
> Please see if you can verify whether the attached fixes it in all the
> branches?
Yes, the attached patch fixes the issue in all the branches.
Thanks for the fix!
Best regards,
Alexander
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus()
@ 2021-06-28 08:38 Amit Kapila <amit.kapila16@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 0 replies; 9+ messages in thread
From: Amit Kapila @ 2021-06-28 08:38 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, Jun 28, 2021 at 12:30 AM Alexander Lakhin <exclusion@gmail.com> wrote:
>
> Hello Amit,
> 26.06.2021 12:57, Amit Kapila wrote:
> >
> >>> I think this should be reproducible on all branches from HEAD till
> >>> v11. Have you tried in any other branch? I'll also try to reproduce
> >>> it.
> >> I've reproduced it on REL_11_STABLE, REL_12_STABLE, REL_13_STABLE, and
> >> master.
> >>
> > Please see if you can verify whether the attached fixes it in all the
> > branches?
> Yes, the attached patch fixes the issue in all the branches.
> Thanks for the fix!
>
Pushed, thanks for the report and all the testing.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2021-06-28 08:38 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-06-24 16:00 BUG #17072: Assert for clogGroupNext failed due to a race condition in TransactionGroupUpdateXidStatus() PG Bug reporting form <noreply@postgresql.org>
2021-06-24 16:15 ` Alexander Lakhin <exclusion@gmail.com>
2021-06-25 10:32 ` vignesh C <vignesh21@gmail.com>
2021-06-25 09:55 ` Amit Kapila <amit.kapila16@gmail.com>
2021-06-25 11:00 ` Alexander Lakhin <exclusion@gmail.com>
2021-06-26 09:57 ` Amit Kapila <amit.kapila16@gmail.com>
2021-06-27 19:00 ` Alexander Lakhin <exclusion@gmail.com>
2021-06-28 08:38 ` Amit Kapila <amit.kapila16@gmail.com>
2021-06-25 10:00 ` Alexander Lakhin <exclusion@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox