pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedCREATE INDEX CONCURRENTLY does not index prepared xact's data
107+ messages / 11 participants
[nested] [flat]
Commits from this thread:
2021-10-24 fdd965d074 0869e53d3a 2e33b43599 dde966efb2 e428699cb3 5141e471b3 db86746fd1 Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
2021-10-24 3cd9c3b921 a5b9a0000e fe5d44a1d3 a9d0a54094 df6158139f 5184932432 560124a37c Fix CREATE INDEX CONCURRENTLY for the newest prepared transactions.
2021-01-30 8a54e12a38 86a5b309c9 be843ce297 d1ab4bf6ed 179775135b d683d6528d 77a0e14f0b Fix CREATE INDEX CONCURRENTLY for simultaneous prepared transactions.
* CREATE INDEX CONCURRENTLY does not index prepared xact's data
@ 2020-12-19 17:13 Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Victor Yegorov <vyegorov@gmail.com>
2020-12-19 17:48 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Victor Yegorov <vyegorov@gmail.com>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 3 replies; 107+ messages in thread
From: Andrey Borodin @ 2020-12-19 17:13 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
Hi hackers!
$subj.
Steps to reproduce:
create extension if not exists amcheck;
create table if not exists t1(i int);
begin;
insert into t1 values(1);
prepare transaction 'x';
create index concurrently i1 on t1(i);
commit prepared 'x';
select bt_index_check('i1', true);
I observe:
NOTICE: heap tuple (1,8) from table "t1" lacks matching index tuple within index "i1"
I expect: awaiting 'x' commit before index is created, correct index after.
This happens because WaitForLockersMultiple() does not take prepared xacts into account. Meanwhile CREATE INDEX CONCURRENTLY expects that locks are dropped only when transaction commit is visible.
This issue affects pg_repack and similar machinery based on CIC.
PFA draft of a fix.
Best regards, Andrey Borodin.
Attachments:
v1-0001-Wait-for-prepared-xacts-in-CREATE-INDEX-CONCURREN.patch (application/octet-stream, 5.1K) download
From 02a3baa7f963d5116b3e4506a0f1dcac2c3c16a9 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 19 Dec 2020 21:53:00 +0500
Subject: [PATCH v1] Wait for prepared xacts in CREATE INDEX CONCURRENTLY locks
infrastructure
---
src/backend/storage/lmgr/lmgr.c | 5 ++---
src/backend/storage/lmgr/lock.c | 33 +++++++++++++++++++++------------
src/include/storage/lock.h | 5 +++++
3 files changed, 28 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 7409de9405..c1e8e76f41 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -904,8 +904,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
- * check for that. Also, prepared xacts are not reported, which is fine
- * since they certainly aren't going to do anything anymore.
+ * check for that. Also, prepared xacts are reported and awaited.
*/
/* Finally wait for each such transaction to complete */
@@ -913,7 +912,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
{
VirtualTransactionId *lockholders = lfirst(lc);
- while (VirtualTransactionIdIsValid(*lockholders))
+ while (VirtualTransactionIdIsValidOrPreparedXact(*lockholders))
{
/* If requested, publish who we're going to wait for. */
if (progress)
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index d86566f455..9681fcabc8 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2904,9 +2904,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
* so use of this function has to be thought about carefully.
*
* Note we never include the current xact's vxid in the result array,
- * since an xact never blocks itself. Also, prepared transactions are
- * ignored, which is a bit more debatable but is appropriate for current
- * uses of the result.
+ * since an xact never blocks itself.
*/
VirtualTransactionId *
GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
@@ -2931,15 +2929,17 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* Allocate memory to store results, and fill with InvalidVXID. We only
- * need enough space for MaxBackends + a terminator, since prepared xacts
- * don't count. InHotStandby allocate once in TopMemoryContext.
+ * need enough space for MaxBackends + max_prepared_xacts + a
+ * terminator. InHotStandby allocate once in TopMemoryContext.
*/
if (InHotStandby)
{
if (vxids == NULL)
vxids = (VirtualTransactionId *)
MemoryContextAlloc(TopMemoryContext,
- sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ sizeof(VirtualTransactionId) * (MaxBackends
+ + max_prepared_xacts
+ + 1));
}
else
vxids = (VirtualTransactionId *)
@@ -3085,10 +3085,10 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* If we see an invalid VXID, then either the xact has already
- * committed (or aborted), or it's a prepared xact. In either
- * case we may ignore it.
+ * committed (or aborted), or it's a prepared xact. We take
+ * prepared xacts into account too.
*/
- if (VirtualTransactionIdIsValid(vxid))
+ if (VirtualTransactionIdIsValidOrPreparedXact(vxid))
{
int i;
@@ -3108,7 +3108,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
LWLockRelease(partitionLock);
- if (count > MaxBackends) /* should never happen */
+ if (count > MaxBackends + max_prepared_xacts) /* should never happen */
elog(PANIC, "too many conflicting locks found");
vxids[count].backendId = InvalidBackendId;
@@ -4461,9 +4461,18 @@ bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
LOCKTAG tag;
- PGPROC *proc;
+ PGPROC *proc = NULL;
- Assert(VirtualTransactionIdIsValid(vxid));
+ Assert(VirtualTransactionIdIsValidOrPreparedXact(vxid));
+
+ if (VirtualTransactionIdIsPreparedXact(vxid))
+ {
+ /* If it's prepared xact - just wait for it */
+ SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
+ (void) LockAcquire(&tag, ShareLock, false, false);
+ LockRelease(&tag, ShareLock, false);
+ return true;
+ }
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 1c3e9c1999..dbd2a32678 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -70,6 +70,11 @@ typedef struct
#define VirtualTransactionIdIsValid(vxid) \
(((vxid).backendId != InvalidBackendId) && \
LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsValidOrPreparedXact(vxid) \
+ (((vxid).backendId == InvalidBackendId) || \
+ LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsPreparedXact(vxid) \
+ ((vxid).backendId == InvalidBackendId)
#define VirtualTransactionIdEquals(vxid1, vxid2) \
((vxid1).backendId == (vxid2).backendId && \
(vxid1).localTransactionId == (vxid2).localTransactionId)
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2020-12-19 17:22 ` Victor Yegorov <vyegorov@gmail.com>
2020-12-19 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2 siblings, 1 reply; 107+ messages in thread
From: Victor Yegorov @ 2020-12-19 17:22 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
сб, 19 дек. 2020 г. в 18:13, Andrey Borodin <x4mmm@yandex-team.ru>:
> Steps to reproduce:
> create extension if not exists amcheck;
> create table if not exists t1(i int);
> begin;
> insert into t1 values(1);
> prepare transaction 'x';
> create index concurrently i1 on t1(i);
> commit prepared 'x';
> select bt_index_check('i1', true);
>
> I observe:
> NOTICE: heap tuple (1,8) from table "t1" lacks matching index tuple
> within index "i1"
> I expect: awaiting 'x' commit before index is created, correct index after.
>
CREATE INDEX CONCURRENTLY isn't supposed to be run inside a transaction?..
--
Victor Yegorov
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Victor Yegorov <vyegorov@gmail.com>
@ 2020-12-19 17:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 0 replies; 107+ messages in thread
From: Andrey Borodin @ 2020-12-19 17:25 UTC (permalink / raw)
To: Victor Yegorov <vyegorov@gmail.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 19 дек. 2020 г., в 22:22, Victor Yegorov <vyegorov@gmail.com> написал(а):
>
> CREATE INDEX CONCURRENTLY isn't supposed to be run inside a transaction?..
CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2020-12-19 17:48 ` Victor Yegorov <vyegorov@gmail.com>
2020-12-19 18:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2 siblings, 1 reply; 107+ messages in thread
From: Victor Yegorov @ 2020-12-19 17:48 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
сб, 19 дек. 2020 г. в 18:13, Andrey Borodin <x4mmm@yandex-team.ru>:
> I observe:
> NOTICE: heap tuple (1,8) from table "t1" lacks matching index tuple
> within index "i1"
> I expect: awaiting 'x' commit before index is created, correct index after.
>
I agree, that behaviour is unexpected. But getting a notice that requires me
to re-create the index some time later is not better (from DBA perspective).
Maybe it'd be better to wait on prepared xacts like on other open ordinary
transactions?
--
Victor Yegorov
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:48 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Victor Yegorov <vyegorov@gmail.com>
@ 2020-12-19 18:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-20 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2020-12-19 18:25 UTC (permalink / raw)
To: Victor Yegorov <vyegorov@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 19 дек. 2020 г., в 22:48, Victor Yegorov <vyegorov@gmail.com> написал(а):
>
> сб, 19 дек. 2020 г. в 18:13, Andrey Borodin <x4mmm@yandex-team.ru>:
> I observe:
> NOTICE: heap tuple (1,8) from table "t1" lacks matching index tuple within index "i1"
> I expect: awaiting 'x' commit before index is created, correct index after.
>
> I agree, that behaviour is unexpected. But getting a notice that requires me
> to re-create the index some time later is not better (from DBA perspective).
>
> Maybe it'd be better to wait on prepared xacts like on other open ordinary transactions?
This is not a real NOTICE. I just used a bit altered amcheck to diagnose the problem. It's an incorrect index. It lacks some tuples. It will not find existing data, failing to provide "read committed" consistency guaranties.
Fix waits for prepared xacts just like any other transaction.
> 19 дек. 2020 г., в 22:57, Tom Lane <tgl@sss.pgh.pa.us> написал(а):
>
>> Meanwhile CREATE INDEX CONCURRENTLY expects that locks are dropped only
>> when transaction commit is visible.
>
> Don't follow your point here --- I'm pretty sure that prepared xacts
> continue to hold their locks.
Uhmm, yes, locks are there. Relation is locked with RowExclusiveLock, but this lock is ignored by WaitForLockers(heaplocktag, ShareLock, true). Locking relation with ShareLock works as expected.
>> PFA draft of a fix.
>
> Haven't you completely broken VirtualXactLock()? Certainly, whether the
> target is a normal or prepared transaction shouldn't alter the meaning
> of the "wait" flag.
You are right, the patch has a bug here.
> In general, I wonder whether WaitForLockersMultiple and GetLockConflicts
> need to gain an additional parameter indicating whether to consider
> prepared xacts. It's not clear to me that their current behavior is wrong
> for all possible uses.
I don't see usages besides indexing stuff. But maybe it worth to analyse each case...
BTW do we need a test for this? Will isolation test be good at checking this?
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:48 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Victor Yegorov <vyegorov@gmail.com>
2020-12-19 18:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2020-12-20 18:10 ` Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 0 replies; 107+ messages in thread
From: Andrey Borodin @ 2020-12-20 18:10 UTC (permalink / raw)
To: Victor Yegorov <vyegorov@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 19 дек. 2020 г., в 23:25, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
>
> BTW do we need a test for this? Will isolation test be good at checking this?
PFA patch set with isolation test for the $subj and fix for VirtualXactLock() bug.
I think I'll register the thread on January CF.
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
v2-0001-Wait-for-prepared-xacts-in-CREATE-INDEX-CONCURREN.patch (application/octet-stream, 5.2K) download
From d86f33a09b556c179e34362b87154ea13a7efd4b Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 19 Dec 2020 21:53:00 +0500
Subject: [PATCH v2 1/2] Wait for prepared xacts in CREATE INDEX CONCURRENTLY
locks infrastructure
---
src/backend/storage/lmgr/lmgr.c | 5 ++---
src/backend/storage/lmgr/lock.c | 35 ++++++++++++++++++++++-----------
src/include/storage/lock.h | 5 +++++
3 files changed, 30 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 7409de9405..c1e8e76f41 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -904,8 +904,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
- * check for that. Also, prepared xacts are not reported, which is fine
- * since they certainly aren't going to do anything anymore.
+ * check for that. Also, prepared xacts are reported and awaited.
*/
/* Finally wait for each such transaction to complete */
@@ -913,7 +912,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
{
VirtualTransactionId *lockholders = lfirst(lc);
- while (VirtualTransactionIdIsValid(*lockholders))
+ while (VirtualTransactionIdIsValidOrPreparedXact(*lockholders))
{
/* If requested, publish who we're going to wait for. */
if (progress)
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index d86566f455..c8cecd6f7a 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2904,9 +2904,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
* so use of this function has to be thought about carefully.
*
* Note we never include the current xact's vxid in the result array,
- * since an xact never blocks itself. Also, prepared transactions are
- * ignored, which is a bit more debatable but is appropriate for current
- * uses of the result.
+ * since an xact never blocks itself.
*/
VirtualTransactionId *
GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
@@ -2931,15 +2929,17 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* Allocate memory to store results, and fill with InvalidVXID. We only
- * need enough space for MaxBackends + a terminator, since prepared xacts
- * don't count. InHotStandby allocate once in TopMemoryContext.
+ * need enough space for MaxBackends + max_prepared_xacts + a
+ * terminator. InHotStandby allocate once in TopMemoryContext.
*/
if (InHotStandby)
{
if (vxids == NULL)
vxids = (VirtualTransactionId *)
MemoryContextAlloc(TopMemoryContext,
- sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ sizeof(VirtualTransactionId) * (MaxBackends
+ + max_prepared_xacts
+ + 1));
}
else
vxids = (VirtualTransactionId *)
@@ -3085,10 +3085,10 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* If we see an invalid VXID, then either the xact has already
- * committed (or aborted), or it's a prepared xact. In either
- * case we may ignore it.
+ * committed (or aborted), or it's a prepared xact. We take
+ * prepared xacts into account too.
*/
- if (VirtualTransactionIdIsValid(vxid))
+ if (VirtualTransactionIdIsValidOrPreparedXact(vxid))
{
int i;
@@ -3108,7 +3108,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
LWLockRelease(partitionLock);
- if (count > MaxBackends) /* should never happen */
+ if (count > MaxBackends + max_prepared_xacts) /* should never happen */
elog(PANIC, "too many conflicting locks found");
vxids[count].backendId = InvalidBackendId;
@@ -4461,9 +4461,20 @@ bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
LOCKTAG tag;
- PGPROC *proc;
+ PGPROC *proc = NULL;
- Assert(VirtualTransactionIdIsValid(vxid));
+ Assert(VirtualTransactionIdIsValidOrPreparedXact(vxid));
+
+ if (VirtualTransactionIdIsPreparedXact(vxid))
+ {
+ LockAcquireResult lar;
+ /* If it's prepared xact - just wait for it */
+ SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_OK)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 1c3e9c1999..dbd2a32678 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -70,6 +70,11 @@ typedef struct
#define VirtualTransactionIdIsValid(vxid) \
(((vxid).backendId != InvalidBackendId) && \
LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsValidOrPreparedXact(vxid) \
+ (((vxid).backendId == InvalidBackendId) || \
+ LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsPreparedXact(vxid) \
+ ((vxid).backendId == InvalidBackendId)
#define VirtualTransactionIdEquals(vxid1, vxid2) \
((vxid1).backendId == (vxid2).backendId && \
(vxid1).localTransactionId == (vxid2).localTransactionId)
--
2.24.3 (Apple Git-128)
=
v2-0002-Add-test-for-CIC-with-prepared-xacts.patch (application/octet-stream, 3.1K) download
From 60aae8cfce867561dda441485bbce816bcb00b5e Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 20 Dec 2020 23:03:01 +0500
Subject: [PATCH v2 2/2] Add test for CIC with prepared xacts
---
src/test/isolation/Makefile | 4 +--
.../expected/prepared-transactions-cic.out | 18 ++++++++++
.../specs/prepared-transactions-cic.spec | 34 +++++++++++++++++++
3 files changed, 54 insertions(+), 2 deletions(-)
create mode 100644 src/test/isolation/expected/prepared-transactions-cic.out
create mode 100644 src/test/isolation/specs/prepared-transactions-cic.spec
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index d23e2cec64..a1c3248eb4 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -67,7 +67,7 @@ check: all
# via TEMP_CONFIG for the check case, or via the postgresql.conf for the
# installcheck case.
installcheck-prepared-txns: all temp-install
- $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
check-prepared-txns: all temp-install
- $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
diff --git a/src/test/isolation/expected/prepared-transactions-cic.out b/src/test/isolation/expected/prepared-transactions-cic.out
new file mode 100644
index 0000000000..043ec3c363
--- /dev/null
+++ b/src/test/isolation/expected/prepared-transactions-cic.out
@@ -0,0 +1,18 @@
+Parsed test spec with 2 sessions
+
+starting permutation: w1 p1 cic2 c1 r2
+step w1: BEGIN; INSERT INTO cic_test VALUES (1);
+step p1: PREPARE TRANSACTION 's1';
+step cic2:
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+
+ERROR: canceling statement due to lock timeout
+step c1: COMMIT PREPARED 's1';
+step r2:
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+
+a
+
+1
diff --git a/src/test/isolation/specs/prepared-transactions-cic.spec b/src/test/isolation/specs/prepared-transactions-cic.spec
new file mode 100644
index 0000000000..65e2c00b4c
--- /dev/null
+++ b/src/test/isolation/specs/prepared-transactions-cic.spec
@@ -0,0 +1,34 @@
+# This test verifies that CREATE INDEX CONCURRENTLY interact with
+# prepared transactions correctly.
+setup
+{
+ CREATE TABLE cic_test (a int);
+}
+
+teardown
+{
+ DROP TABLE cic_test;
+}
+
+
+# Sessions for CREATE INDEX CONCURRENTLY test
+session "s1"
+step "w1" { BEGIN; INSERT INTO cic_test VALUES (1); }
+step "p1" { PREPARE TRANSACTION 's1'; }
+step "c1" { COMMIT PREPARED 's1'; }
+
+session "s2"
+setup { SET lock_timeout = 10; }
+step "cic2"
+{
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+}
+step "r2"
+{
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+}
+
+
+permutation "w1" "p1" "cic2" "c1" "r2"
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2020-12-19 17:57 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2 siblings, 1 reply; 107+ messages in thread
From: Tom Lane @ 2020-12-19 17:57 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: pgsql-bugs@lists.postgresql.org
Andrey Borodin <x4mmm@yandex-team.ru> writes:
> This happens because WaitForLockersMultiple() does not take prepared
> xacts into account.
Ugh, clearly an oversight.
> Meanwhile CREATE INDEX CONCURRENTLY expects that locks are dropped only
> when transaction commit is visible.
Don't follow your point here --- I'm pretty sure that prepared xacts
continue to hold their locks.
> PFA draft of a fix.
Haven't you completely broken VirtualXactLock()? Certainly, whether the
target is a normal or prepared transaction shouldn't alter the meaning
of the "wait" flag.
In general, I wonder whether WaitForLockersMultiple and GetLockConflicts
need to gain an additional parameter indicating whether to consider
prepared xacts. It's not clear to me that their current behavior is wrong
for all possible uses.
regards, tom lane
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
@ 2020-12-21 05:40 ` Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-23 07:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 107+ messages in thread
From: Michael Paquier @ 2020-12-21 05:40 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; pgsql-bugs@lists.postgresql.org
On Sat, Dec 19, 2020 at 12:57:41PM -0500, Tom Lane wrote:
> Andrey Borodin <x4mmm@yandex-team.ru> writes:
>> This happens because WaitForLockersMultiple() does not take prepared
>> xacts into account.
>
> Ugh, clearly an oversight.
This looks to be the case since 295e639 where virtual XIDs have been
introduced. So this is an old bug.
> Don't follow your point here --- I'm pretty sure that prepared xacts
> continue to hold their locks.
Yes, that's what I recall as well.
> Haven't you completely broken VirtualXactLock()? Certainly, whether the
> target is a normal or prepared transaction shouldn't alter the meaning
> of the "wait" flag.
Yep.
> In general, I wonder whether WaitForLockersMultiple and GetLockConflicts
> need to gain an additional parameter indicating whether to consider
> prepared xacts. It's not clear to me that their current behavior is wrong
> for all possible uses.
WaitForLockers is used only by REINDEX and CREATE/DROP CONCURRENTLY,
where it seems to me we need to care about all the cases related to
concurrent build, validation and index drop. The other caller of
GetLockConflicts() is for conflict resolution in standbys where it is
fine to ignore 2PC transactions as these cannot be cancelled. So I
agree that we are going to need more control with a new option
argument to be able to control if 2PC transactions are ignored or
not.
Hmm. The approach taken by the patch looks to be back-patchable.
Based on the lack of complaints on the matter, we could consider
instead putting an error in WaitForLockersMultiple() if there is at
least one numPrepXact which would at least avoid inconsistent data.
But I don't think what's proposed here is bad either.
VirtualTransactionIdIsValidOrPreparedXact() is confusing IMO, knowing
that VirtualTransactionIdIsPreparedXact() combined with
LocalTransactionIdIsValid() would be enough to do the job.
- Assert(VirtualTransactionIdIsValid(vxid));
+ Assert(VirtualTransactionIdIsValidOrPreparedXact(vxid));
+
+ if (VirtualTransactionIdIsPreparedXact(vxid))
[...]
#define VirtualTransactionIdIsPreparedXact(vxid) \
((vxid).backendId == InvalidBackendId)
This would allow the case where backendId and localTransactionId are
both invalid. So it would be better to also check in
VirtualTransactionIdIsPreparedXact() that the XID is not invalid, no?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
@ 2020-12-21 07:24 ` Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2020-12-21 07:24 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 21 дек. 2020 г., в 10:40, Michael Paquier <michael@paquier.xyz> написал(а):
>
>> In general, I wonder whether WaitForLockersMultiple and GetLockConflicts
>> need to gain an additional parameter indicating whether to consider
>> prepared xacts. It's not clear to me that their current behavior is wrong
>> for all possible uses.
>
> WaitForLockers is used only by REINDEX and CREATE/DROP CONCURRENTLY,
> where it seems to me we need to care about all the cases related to
> concurrent build, validation and index drop. The other caller of
> GetLockConflicts() is for conflict resolution in standbys where it is
> fine to ignore 2PC transactions as these cannot be cancelled.
I don't think that fact that we cannot cancel transaction is sufficient here to ignore prepared transaction. I think there should not exist any prepared transaction that we need to cancel in standby conflict resolution. And if it exists - it's a sign of corruption, we could emit warning or something like that.
But I'm really not an expert here, just a common sense that prepared transaction is just like regular transaction that survives crash. If we wait for any transaction - probably we should wait for prepared too. I'm not insisting on anything though.
> So I
> agree that we are going to need more control with a new option
> argument to be able to control if 2PC transactions are ignored or
> not.
>
> Hmm. The approach taken by the patch looks to be back-patchable.
> Based on the lack of complaints on the matter, we could consider
> instead putting an error in WaitForLockersMultiple() if there is at
> least one numPrepXact which would at least avoid inconsistent data.
> But I don't think what's proposed here is bad either.
>
> VirtualTransactionIdIsValidOrPreparedXact() is confusing IMO, knowing
> that VirtualTransactionIdIsPreparedXact() combined with
> LocalTransactionIdIsValid() would be enough to do the job.
>
> - Assert(VirtualTransactionIdIsValid(vxid));
> + Assert(VirtualTransactionIdIsValidOrPreparedXact(vxid));
> +
> + if (VirtualTransactionIdIsPreparedXact(vxid))
> [...]
> #define VirtualTransactionIdIsPreparedXact(vxid) \
> ((vxid).backendId == InvalidBackendId)
> This would allow the case where backendId and localTransactionId are
> both invalid. So it would be better to also check in
> VirtualTransactionIdIsPreparedXact() that the XID is not invalid, no?
Seems valid. Removed VirtualTransactionIdIsValidOrPreparedXact() from patch.
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
v3-0001-Wait-for-prepared-xacts-in-CREATE-INDEX-CONCURREN.patch (application/octet-stream, 5.2K) download
From 589c0e98ae913a61d7f2833091a89d0f4e1ce556 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 19 Dec 2020 21:53:00 +0500
Subject: [PATCH v3 1/2] Wait for prepared xacts in CREATE INDEX CONCURRENTLY
locks infrastructure
---
src/backend/storage/lmgr/lmgr.c | 5 ++---
src/backend/storage/lmgr/lock.c | 37 ++++++++++++++++++++++-----------
src/include/storage/lock.h | 5 +++++
3 files changed, 32 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 7409de9405..c1e8e76f41 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -904,8 +904,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
- * check for that. Also, prepared xacts are not reported, which is fine
- * since they certainly aren't going to do anything anymore.
+ * check for that. Also, prepared xacts are reported and awaited.
*/
/* Finally wait for each such transaction to complete */
@@ -913,7 +912,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
{
VirtualTransactionId *lockholders = lfirst(lc);
- while (VirtualTransactionIdIsValid(*lockholders))
+ while (VirtualTransactionIdIsValidOrPreparedXact(*lockholders))
{
/* If requested, publish who we're going to wait for. */
if (progress)
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index d86566f455..d870396504 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2904,9 +2904,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
* so use of this function has to be thought about carefully.
*
* Note we never include the current xact's vxid in the result array,
- * since an xact never blocks itself. Also, prepared transactions are
- * ignored, which is a bit more debatable but is appropriate for current
- * uses of the result.
+ * since an xact never blocks itself.
*/
VirtualTransactionId *
GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
@@ -2931,15 +2929,17 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* Allocate memory to store results, and fill with InvalidVXID. We only
- * need enough space for MaxBackends + a terminator, since prepared xacts
- * don't count. InHotStandby allocate once in TopMemoryContext.
+ * need enough space for MaxBackends + max_prepared_xacts + a
+ * terminator. InHotStandby allocate once in TopMemoryContext.
*/
if (InHotStandby)
{
if (vxids == NULL)
vxids = (VirtualTransactionId *)
MemoryContextAlloc(TopMemoryContext,
- sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ sizeof(VirtualTransactionId) * (MaxBackends
+ + max_prepared_xacts
+ + 1));
}
else
vxids = (VirtualTransactionId *)
@@ -3085,10 +3085,11 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* If we see an invalid VXID, then either the xact has already
- * committed (or aborted), or it's a prepared xact. In either
- * case we may ignore it.
+ * committed (or aborted), or it's a prepared xact. We take
+ * prepared xacts into account too.
*/
- if (VirtualTransactionIdIsValid(vxid))
+ if (VirtualTransactionIdIsValid(vxid)
+ || VirtualTransactionIdIsPreparedXact(vxid))
{
int i;
@@ -3108,7 +3109,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
LWLockRelease(partitionLock);
- if (count > MaxBackends) /* should never happen */
+ if (count > MaxBackends + max_prepared_xacts) /* should never happen */
elog(PANIC, "too many conflicting locks found");
vxids[count].backendId = InvalidBackendId;
@@ -4461,9 +4462,21 @@ bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
LOCKTAG tag;
- PGPROC *proc;
+ PGPROC *proc = NULL;
- Assert(VirtualTransactionIdIsValid(vxid));
+ Assert(VirtualTransactionIdIsValid(vxid) ||
+ VirtualTransactionIdIsPreparedXact(vxid));
+
+ if (VirtualTransactionIdIsPreparedXact(vxid))
+ {
+ LockAcquireResult lar;
+ /* If it's prepared xact - just wait for it */
+ SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_OK)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 1c3e9c1999..dbd2a32678 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -70,6 +70,11 @@ typedef struct
#define VirtualTransactionIdIsValid(vxid) \
(((vxid).backendId != InvalidBackendId) && \
LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsValidOrPreparedXact(vxid) \
+ (((vxid).backendId == InvalidBackendId) || \
+ LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsPreparedXact(vxid) \
+ ((vxid).backendId == InvalidBackendId)
#define VirtualTransactionIdEquals(vxid1, vxid2) \
((vxid1).backendId == (vxid2).backendId && \
(vxid1).localTransactionId == (vxid2).localTransactionId)
--
2.24.3 (Apple Git-128)
=
v3-0002-Add-test-for-CIC-with-prepared-xacts.patch (application/octet-stream, 3.1K) download
From ad5a84fe796753fce7f3ec50c538b0854ea68ddd Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 20 Dec 2020 23:03:01 +0500
Subject: [PATCH v3 2/2] Add test for CIC with prepared xacts
---
src/test/isolation/Makefile | 4 +--
.../expected/prepared-transactions-cic.out | 18 ++++++++++
.../specs/prepared-transactions-cic.spec | 34 +++++++++++++++++++
3 files changed, 54 insertions(+), 2 deletions(-)
create mode 100644 src/test/isolation/expected/prepared-transactions-cic.out
create mode 100644 src/test/isolation/specs/prepared-transactions-cic.spec
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index d23e2cec64..a1c3248eb4 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -67,7 +67,7 @@ check: all
# via TEMP_CONFIG for the check case, or via the postgresql.conf for the
# installcheck case.
installcheck-prepared-txns: all temp-install
- $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
check-prepared-txns: all temp-install
- $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
diff --git a/src/test/isolation/expected/prepared-transactions-cic.out b/src/test/isolation/expected/prepared-transactions-cic.out
new file mode 100644
index 0000000000..043ec3c363
--- /dev/null
+++ b/src/test/isolation/expected/prepared-transactions-cic.out
@@ -0,0 +1,18 @@
+Parsed test spec with 2 sessions
+
+starting permutation: w1 p1 cic2 c1 r2
+step w1: BEGIN; INSERT INTO cic_test VALUES (1);
+step p1: PREPARE TRANSACTION 's1';
+step cic2:
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+
+ERROR: canceling statement due to lock timeout
+step c1: COMMIT PREPARED 's1';
+step r2:
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+
+a
+
+1
diff --git a/src/test/isolation/specs/prepared-transactions-cic.spec b/src/test/isolation/specs/prepared-transactions-cic.spec
new file mode 100644
index 0000000000..65e2c00b4c
--- /dev/null
+++ b/src/test/isolation/specs/prepared-transactions-cic.spec
@@ -0,0 +1,34 @@
+# This test verifies that CREATE INDEX CONCURRENTLY interact with
+# prepared transactions correctly.
+setup
+{
+ CREATE TABLE cic_test (a int);
+}
+
+teardown
+{
+ DROP TABLE cic_test;
+}
+
+
+# Sessions for CREATE INDEX CONCURRENTLY test
+session "s1"
+step "w1" { BEGIN; INSERT INTO cic_test VALUES (1); }
+step "p1" { PREPARE TRANSACTION 's1'; }
+step "c1" { COMMIT PREPARED 's1'; }
+
+session "s2"
+setup { SET lock_timeout = 10; }
+step "cic2"
+{
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+}
+step "r2"
+{
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+}
+
+
+permutation "w1" "p1" "cic2" "c1" "r2"
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2020-12-21 08:19 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2020-12-21 08:19 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 21 дек. 2020 г., в 12:24, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
>
> Seems valid. Removed VirtualTransactionIdIsValidOrPreparedXact() from patch.
Sorry for the noise, removal was not complete.
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
v4-0002-Add-test-for-CIC-with-prepared-xacts.patch (application/octet-stream, 3.1K) download
From 857671ff70304c3a09c9428f29d842e9f5111a81 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 20 Dec 2020 23:03:01 +0500
Subject: [PATCH v4 2/2] Add test for CIC with prepared xacts
---
src/test/isolation/Makefile | 4 +--
.../expected/prepared-transactions-cic.out | 18 ++++++++++
.../specs/prepared-transactions-cic.spec | 34 +++++++++++++++++++
3 files changed, 54 insertions(+), 2 deletions(-)
create mode 100644 src/test/isolation/expected/prepared-transactions-cic.out
create mode 100644 src/test/isolation/specs/prepared-transactions-cic.spec
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index d23e2cec64..a1c3248eb4 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -67,7 +67,7 @@ check: all
# via TEMP_CONFIG for the check case, or via the postgresql.conf for the
# installcheck case.
installcheck-prepared-txns: all temp-install
- $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
check-prepared-txns: all temp-install
- $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
diff --git a/src/test/isolation/expected/prepared-transactions-cic.out b/src/test/isolation/expected/prepared-transactions-cic.out
new file mode 100644
index 0000000000..043ec3c363
--- /dev/null
+++ b/src/test/isolation/expected/prepared-transactions-cic.out
@@ -0,0 +1,18 @@
+Parsed test spec with 2 sessions
+
+starting permutation: w1 p1 cic2 c1 r2
+step w1: BEGIN; INSERT INTO cic_test VALUES (1);
+step p1: PREPARE TRANSACTION 's1';
+step cic2:
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+
+ERROR: canceling statement due to lock timeout
+step c1: COMMIT PREPARED 's1';
+step r2:
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+
+a
+
+1
diff --git a/src/test/isolation/specs/prepared-transactions-cic.spec b/src/test/isolation/specs/prepared-transactions-cic.spec
new file mode 100644
index 0000000000..65e2c00b4c
--- /dev/null
+++ b/src/test/isolation/specs/prepared-transactions-cic.spec
@@ -0,0 +1,34 @@
+# This test verifies that CREATE INDEX CONCURRENTLY interact with
+# prepared transactions correctly.
+setup
+{
+ CREATE TABLE cic_test (a int);
+}
+
+teardown
+{
+ DROP TABLE cic_test;
+}
+
+
+# Sessions for CREATE INDEX CONCURRENTLY test
+session "s1"
+step "w1" { BEGIN; INSERT INTO cic_test VALUES (1); }
+step "p1" { PREPARE TRANSACTION 's1'; }
+step "c1" { COMMIT PREPARED 's1'; }
+
+session "s2"
+setup { SET lock_timeout = 10; }
+step "cic2"
+{
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+}
+step "r2"
+{
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+}
+
+
+permutation "w1" "p1" "cic2" "c1" "r2"
--
2.24.3 (Apple Git-128)
=
v4-0001-Wait-for-prepared-xacts-in-CREATE-INDEX-CONCURREN.patch (application/octet-stream, 5.1K) download
From 55f117c37ce6cac8304db225725bda5e4c82e80d Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 19 Dec 2020 21:53:00 +0500
Subject: [PATCH v4 1/2] Wait for prepared xacts in CREATE INDEX CONCURRENTLY
locks infrastructure
---
src/backend/storage/lmgr/lmgr.c | 6 +++---
src/backend/storage/lmgr/lock.c | 37 ++++++++++++++++++++++-----------
src/include/storage/lock.h | 2 ++
3 files changed, 30 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 7409de9405..01487930fd 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -904,8 +904,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
- * check for that. Also, prepared xacts are not reported, which is fine
- * since they certainly aren't going to do anything anymore.
+ * check for that. Also, prepared xacts are reported and awaited.
*/
/* Finally wait for each such transaction to complete */
@@ -913,7 +912,8 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
{
VirtualTransactionId *lockholders = lfirst(lc);
- while (VirtualTransactionIdIsValid(*lockholders))
+ while (VirtualTransactionIdIsValid(*lockholders) ||
+ VirtualTransactionIdIsPreparedXact(*lockholders))
{
/* If requested, publish who we're going to wait for. */
if (progress)
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index d86566f455..d870396504 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2904,9 +2904,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
* so use of this function has to be thought about carefully.
*
* Note we never include the current xact's vxid in the result array,
- * since an xact never blocks itself. Also, prepared transactions are
- * ignored, which is a bit more debatable but is appropriate for current
- * uses of the result.
+ * since an xact never blocks itself.
*/
VirtualTransactionId *
GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
@@ -2931,15 +2929,17 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* Allocate memory to store results, and fill with InvalidVXID. We only
- * need enough space for MaxBackends + a terminator, since prepared xacts
- * don't count. InHotStandby allocate once in TopMemoryContext.
+ * need enough space for MaxBackends + max_prepared_xacts + a
+ * terminator. InHotStandby allocate once in TopMemoryContext.
*/
if (InHotStandby)
{
if (vxids == NULL)
vxids = (VirtualTransactionId *)
MemoryContextAlloc(TopMemoryContext,
- sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ sizeof(VirtualTransactionId) * (MaxBackends
+ + max_prepared_xacts
+ + 1));
}
else
vxids = (VirtualTransactionId *)
@@ -3085,10 +3085,11 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* If we see an invalid VXID, then either the xact has already
- * committed (or aborted), or it's a prepared xact. In either
- * case we may ignore it.
+ * committed (or aborted), or it's a prepared xact. We take
+ * prepared xacts into account too.
*/
- if (VirtualTransactionIdIsValid(vxid))
+ if (VirtualTransactionIdIsValid(vxid)
+ || VirtualTransactionIdIsPreparedXact(vxid))
{
int i;
@@ -3108,7 +3109,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
LWLockRelease(partitionLock);
- if (count > MaxBackends) /* should never happen */
+ if (count > MaxBackends + max_prepared_xacts) /* should never happen */
elog(PANIC, "too many conflicting locks found");
vxids[count].backendId = InvalidBackendId;
@@ -4461,9 +4462,21 @@ bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
LOCKTAG tag;
- PGPROC *proc;
+ PGPROC *proc = NULL;
- Assert(VirtualTransactionIdIsValid(vxid));
+ Assert(VirtualTransactionIdIsValid(vxid) ||
+ VirtualTransactionIdIsPreparedXact(vxid));
+
+ if (VirtualTransactionIdIsPreparedXact(vxid))
+ {
+ LockAcquireResult lar;
+ /* If it's prepared xact - just wait for it */
+ SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_OK)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 1c3e9c1999..cedb9d6d2a 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -70,6 +70,8 @@ typedef struct
#define VirtualTransactionIdIsValid(vxid) \
(((vxid).backendId != InvalidBackendId) && \
LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsPreparedXact(vxid) \
+ ((vxid).backendId == InvalidBackendId)
#define VirtualTransactionIdEquals(vxid1, vxid2) \
((vxid1).backendId == (vxid2).backendId && \
(vxid1).localTransactionId == (vxid2).localTransactionId)
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-01-17 07:24 ` Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-01-17 07:24 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Mon, Dec 21, 2020 at 12:24:55PM +0500, Andrey Borodin wrote:
> > 21 дек. 2020 г., в 10:40, Michael Paquier <michael@paquier.xyz> написал(а):
> >> In general, I wonder whether WaitForLockersMultiple and GetLockConflicts
> >> need to gain an additional parameter indicating whether to consider
> >> prepared xacts. It's not clear to me that their current behavior is wrong
> >> for all possible uses.
> >
> > WaitForLockers is used only by REINDEX and CREATE/DROP CONCURRENTLY,
> > where it seems to me we need to care about all the cases related to
> > concurrent build, validation and index drop. The other caller of
> > GetLockConflicts() is for conflict resolution in standbys where it is
> > fine to ignore 2PC transactions as these cannot be cancelled.
>
> I don't think that fact that we cannot cancel transaction is sufficient here to ignore prepared transaction. I think there should not exist any prepared transaction that we need to cancel in standby conflict resolution. And if it exists - it's a sign of corruption, we could emit warning or something like that.
Agreed. Based on src/backend/storage/lmgr/README section "Locking during Hot
Standby" and RecoverPreparedTransactions(), prepared-transaction PGPROC
entries get locks at end of recovery, not earlier. The conflict resolution
callers won't see prepared-transaction locks, and they could assert that,
raise an error, or just assume that implicitly.
> But I'm really not an expert here, just a common sense that prepared transaction is just like regular transaction that survives crash. If we wait for any transaction - probably we should wait for prepared too. I'm not insisting on anything though.
In the startup process of a standby, waiting for a primary-side transaction
(regular or prepared) would succeed instantly, rendering the wait meaningless.
I'm not too worried about it; several parts of the system would change if
these standby-side lock invariants ceased to hold.
On Mon, Dec 21, 2020 at 01:19:59PM +0500, Andrey Borodin wrote:
> --- /dev/null
> +++ b/src/test/isolation/expected/prepared-transactions-cic.out
> @@ -0,0 +1,18 @@
> +Parsed test spec with 2 sessions
> +
> +starting permutation: w1 p1 cic2 c1 r2
> +step w1: BEGIN; INSERT INTO cic_test VALUES (1);
> +step p1: PREPARE TRANSACTION 's1';
> +step cic2:
> + CREATE INDEX CONCURRENTLY on cic_test(a);
> +
> +ERROR: canceling statement due to lock timeout
I wondered if a slow server could ever print "<waiting ...>" in here. It
can't; this is fine. pg_isolation_test_session_is_blocked() never reports the
prepared transaction, because that transaction's locks have no pid associated.
> --- a/src/backend/storage/lmgr/lock.c
> +++ b/src/backend/storage/lmgr/lock.c
> @@ -2931,15 +2929,17 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
>
> /*
> * Allocate memory to store results, and fill with InvalidVXID. We only
> - * need enough space for MaxBackends + a terminator, since prepared xacts
> - * don't count. InHotStandby allocate once in TopMemoryContext.
> + * need enough space for MaxBackends + max_prepared_xacts + a
> + * terminator. InHotStandby allocate once in TopMemoryContext.
> */
> if (InHotStandby)
> {
> if (vxids == NULL)
> vxids = (VirtualTransactionId *)
> MemoryContextAlloc(TopMemoryContext,
> - sizeof(VirtualTransactionId) * (MaxBackends + 1));
> + sizeof(VirtualTransactionId) * (MaxBackends
> + + max_prepared_xacts
> + + 1));
PostgreSQL typically puts the operator before the newline. Also, please note
the whitespace error that "git diff --check origin/master" reports.
> }
> else
> vxids = (VirtualTransactionId *)
This is updating only the InHotStandby branch. Both branches need the change.
> @@ -4461,9 +4462,21 @@ bool
> VirtualXactLock(VirtualTransactionId vxid, bool wait)
> {
> LOCKTAG tag;
> - PGPROC *proc;
> + PGPROC *proc = NULL;
>
> - Assert(VirtualTransactionIdIsValid(vxid));
> + Assert(VirtualTransactionIdIsValid(vxid) ||
> + VirtualTransactionIdIsPreparedXact(vxid));
> +
> + if (VirtualTransactionIdIsPreparedXact(vxid))
> + {
> + LockAcquireResult lar;
> + /* If it's prepared xact - just wait for it */
> + SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
> + lar = LockAcquire(&tag, ShareLock, false, !wait);
> + if (lar == LOCKACQUIRE_OK)
This should instead test "!= LOCKACQUIRE_NOT_AVAIL", lest
LOCKACQUIRE_ALREADY_HELD happen. (It perhaps can't happen, but skipping the
LockRelease() would be wrong if it did.)
> + LockRelease(&tag, ShareLock, false);
> + return lar != LOCKACQUIRE_NOT_AVAIL;
> + }
>
> SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
>
> diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
> index 1c3e9c1999..cedb9d6d2a 100644
> --- a/src/include/storage/lock.h
> +++ b/src/include/storage/lock.h
> @@ -70,6 +70,8 @@ typedef struct
> #define VirtualTransactionIdIsValid(vxid) \
> (((vxid).backendId != InvalidBackendId) && \
> LocalTransactionIdIsValid((vxid).localTransactionId))
> +#define VirtualTransactionIdIsPreparedXact(vxid) \
> + ((vxid).backendId == InvalidBackendId)
Your patch is introducing VirtualTransactionId values that represent prepared
xacts, and it has VirtualTransactionIdIsValid() return false for those values.
Let's make VirtualTransactionIdIsValid() return true for them, since they're
as meaningful as any other value. The GetLockConflicts() header comment says
"The result array is palloc'd and is terminated with an invalid VXID." Patch
v4 falsifies that comment. The array can contain many of these new "invalid"
VXIDs, and an all-zeroes VXID terminates the array. Rather than change the
comment, let's change VirtualTransactionIdIsValid() to render the comment true
again. Most (perhaps all) VirtualTransactionIdIsValid() callers won't need to
distinguish the prepared-transaction case.
An alternative to redefining VXID this way would be to have some new type,
each instance of which holds either a valid VXID or a valid
prepared-transaction XID. That alternative feels inferior to me, though.
What do you think?
Thanks,
nm
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-01-22 05:44 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-22 05:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 2 replies; 107+ messages in thread
From: Andrey Borodin @ 2021-01-22 05:44 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
Thanks for looking into this!
> 17 янв. 2021 г., в 12:24, Noah Misch <noah@leadboat.com> написал(а):
>
>> --- a/src/backend/storage/lmgr/lock.c
>> +++ b/src/backend/storage/lmgr/lock.c
>> @@ -2931,15 +2929,17 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
>>
>> /*
>> * Allocate memory to store results, and fill with InvalidVXID. We only
>> - * need enough space for MaxBackends + a terminator, since prepared xacts
>> - * don't count. InHotStandby allocate once in TopMemoryContext.
>> + * need enough space for MaxBackends + max_prepared_xacts + a
>> + * terminator. InHotStandby allocate once in TopMemoryContext.
>> */
>> if (InHotStandby)
>> {
>> if (vxids == NULL)
>> vxids = (VirtualTransactionId *)
>> MemoryContextAlloc(TopMemoryContext,
>> - sizeof(VirtualTransactionId) * (MaxBackends + 1));
>> + sizeof(VirtualTransactionId) * (MaxBackends
>> + + max_prepared_xacts
>> + + 1));
>
> PostgreSQL typically puts the operator before the newline. Also, please note
> the whitespace error that "git diff --check origin/master" reports.
Fixed.
>
>> }
>> else
>> vxids = (VirtualTransactionId *)
>
> This is updating only the InHotStandby branch. Both branches need the change.
Fixed.
>
>> @@ -4461,9 +4462,21 @@ bool
>> VirtualXactLock(VirtualTransactionId vxid, bool wait)
>> {
>> LOCKTAG tag;
>> - PGPROC *proc;
>> + PGPROC *proc = NULL;
>>
>> - Assert(VirtualTransactionIdIsValid(vxid));
>> + Assert(VirtualTransactionIdIsValid(vxid) ||
>> + VirtualTransactionIdIsPreparedXact(vxid));
>> +
>> + if (VirtualTransactionIdIsPreparedXact(vxid))
>> + {
>> + LockAcquireResult lar;
>> + /* If it's prepared xact - just wait for it */
>> + SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
>> + lar = LockAcquire(&tag, ShareLock, false, !wait);
>> + if (lar == LOCKACQUIRE_OK)
>
> This should instead test "!= LOCKACQUIRE_NOT_AVAIL", lest
> LOCKACQUIRE_ALREADY_HELD happen. (It perhaps can't happen, but skipping the
> LockRelease() would be wrong if it did.)
I think that code that successfully acquired lock should release it. Other way we risk to release someone's else lock held for a reason. We only acquire lock to release it instantly anyway.
>
>> + LockRelease(&tag, ShareLock, false);
>> + return lar != LOCKACQUIRE_NOT_AVAIL;
>> + }
>>
>> SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
>>
>> diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
>> index 1c3e9c1999..cedb9d6d2a 100644
>> --- a/src/include/storage/lock.h
>> +++ b/src/include/storage/lock.h
>> @@ -70,6 +70,8 @@ typedef struct
>> #define VirtualTransactionIdIsValid(vxid) \
>> (((vxid).backendId != InvalidBackendId) && \
>> LocalTransactionIdIsValid((vxid).localTransactionId))
>> +#define VirtualTransactionIdIsPreparedXact(vxid) \
>> + ((vxid).backendId == InvalidBackendId)
>
> Your patch is introducing VirtualTransactionId values that represent prepared
> xacts, and it has VirtualTransactionIdIsValid() return false for those values.
> Let's make VirtualTransactionIdIsValid() return true for them, since they're
> as meaningful as any other value. The GetLockConflicts() header comment says
> "The result array is palloc'd and is terminated with an invalid VXID." Patch
> v4 falsifies that comment. The array can contain many of these new "invalid"
> VXIDs, and an all-zeroes VXID terminates the array. Rather than change the
> comment, let's change VirtualTransactionIdIsValid() to render the comment true
> again. Most (perhaps all) VirtualTransactionIdIsValid() callers won't need to
> distinguish the prepared-transaction case.
Makes sense, fixed. I was afraid that there's something I'm not aware about. I've iterated over VirtualTransactionIdIsValid() calls and did not find suspicious cases.
> An alternative to redefining VXID this way would be to have some new type,
> each instance of which holds either a valid VXID or a valid
> prepared-transaction XID. That alternative feels inferior to me, though.
> What do you think?
I think we should not call valid vxids "invalid".
By the way maybe rename "check-prepared-txns" to "check-prepared-xacts" for consistency?
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
v5-0001-Wait-for-prepared-xacts-in-CREATE-INDEX-CONCURREN.patch (application/octet-stream, 5.6K) download
From 30ec43e1054e4fae47f9221bb834281abb7fc5ac Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 19 Dec 2020 21:53:00 +0500
Subject: [PATCH v5 1/2] Wait for prepared xacts in CREATE INDEX CONCURRENTLY
locks infrastructure
---
src/backend/storage/lmgr/lmgr.c | 6 ++---
src/backend/storage/lmgr/lock.c | 39 ++++++++++++++++++++++-----------
src/include/storage/lock.h | 5 +++--
3 files changed, 32 insertions(+), 18 deletions(-)
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 7407672c9d..07f75d8e06 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -904,8 +904,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
- * check for that. Also, prepared xacts are not reported, which is fine
- * since they certainly aren't going to do anything anymore.
+ * check for that. Also, prepared xacts are reported and awaited.
*/
/* Finally wait for each such transaction to complete */
@@ -913,7 +912,8 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
{
VirtualTransactionId *lockholders = lfirst(lc);
- while (VirtualTransactionIdIsValid(*lockholders))
+ while (VirtualTransactionIdIsValid(*lockholders) ||
+ VirtualTransactionIdIsPreparedXact(*lockholders))
{
/* If requested, publish who we're going to wait for. */
if (progress)
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 20e50247ea..6435e28422 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2903,9 +2903,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
* so use of this function has to be thought about carefully.
*
* Note we never include the current xact's vxid in the result array,
- * since an xact never blocks itself. Also, prepared transactions are
- * ignored, which is a bit more debatable but is appropriate for current
- * uses of the result.
+ * since an xact never blocks itself.
*/
VirtualTransactionId *
GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
@@ -2930,19 +2928,21 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* Allocate memory to store results, and fill with InvalidVXID. We only
- * need enough space for MaxBackends + a terminator, since prepared xacts
- * don't count. InHotStandby allocate once in TopMemoryContext.
+ * need enough space for MaxBackends + max_prepared_xacts + a
+ * terminator. InHotStandby allocate once in TopMemoryContext.
*/
if (InHotStandby)
{
if (vxids == NULL)
vxids = (VirtualTransactionId *)
MemoryContextAlloc(TopMemoryContext,
- sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ sizeof(VirtualTransactionId) * (MaxBackends +
+ max_prepared_xacts + 1));
}
else
vxids = (VirtualTransactionId *)
- palloc0(sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ palloc0(sizeof(VirtualTransactionId) * (MaxBackends +
+ max_prepared_xacts + 1));
/* Compute hash code and partition lock, and look up conflicting modes. */
hashcode = LockTagHashCode(locktag);
@@ -3084,10 +3084,11 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* If we see an invalid VXID, then either the xact has already
- * committed (or aborted), or it's a prepared xact. In either
- * case we may ignore it.
+ * committed (or aborted), or it's a prepared xact. We take
+ * prepared xacts into account too.
*/
- if (VirtualTransactionIdIsValid(vxid))
+ if (VirtualTransactionIdIsValid(vxid)
+ || VirtualTransactionIdIsPreparedXact(vxid))
{
int i;
@@ -3107,7 +3108,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
LWLockRelease(partitionLock);
- if (count > MaxBackends) /* should never happen */
+ if (count > MaxBackends + max_prepared_xacts) /* should never happen */
elog(PANIC, "too many conflicting locks found");
vxids[count].backendId = InvalidBackendId;
@@ -4460,9 +4461,21 @@ bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
LOCKTAG tag;
- PGPROC *proc;
+ PGPROC *proc = NULL;
- Assert(VirtualTransactionIdIsValid(vxid));
+ Assert(VirtualTransactionIdIsValid(vxid) ||
+ VirtualTransactionIdIsPreparedXact(vxid));
+
+ if (VirtualTransactionIdIsPreparedXact(vxid))
+ {
+ LockAcquireResult lar;
+ /* If it's prepared xact - just wait for it */
+ SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_OK)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 2e6ef174e9..221f23e5a6 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -68,8 +68,9 @@ typedef struct
#define InvalidLocalTransactionId 0
#define LocalTransactionIdIsValid(lxid) ((lxid) != InvalidLocalTransactionId)
#define VirtualTransactionIdIsValid(vxid) \
- (((vxid).backendId != InvalidBackendId) && \
- LocalTransactionIdIsValid((vxid).localTransactionId))
+ (LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsPreparedXact(vxid) \
+ ((vxid).backendId == InvalidBackendId)
#define VirtualTransactionIdEquals(vxid1, vxid2) \
((vxid1).backendId == (vxid2).backendId && \
(vxid1).localTransactionId == (vxid2).localTransactionId)
--
2.24.3 (Apple Git-128)
=
v5-0002-Add-test-for-CIC-with-prepared-xacts.patch (application/octet-stream, 3.1K) download
From 97c797f4cafcd071a3c2aad368b1fd666dbcc3fa Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 20 Dec 2020 23:03:01 +0500
Subject: [PATCH v5 2/2] Add test for CIC with prepared xacts
---
src/test/isolation/Makefile | 4 +--
.../expected/prepared-transactions-cic.out | 18 ++++++++++
.../specs/prepared-transactions-cic.spec | 34 +++++++++++++++++++
3 files changed, 54 insertions(+), 2 deletions(-)
create mode 100644 src/test/isolation/expected/prepared-transactions-cic.out
create mode 100644 src/test/isolation/specs/prepared-transactions-cic.spec
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index d23e2cec64..a1c3248eb4 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -67,7 +67,7 @@ check: all
# via TEMP_CONFIG for the check case, or via the postgresql.conf for the
# installcheck case.
installcheck-prepared-txns: all temp-install
- $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
check-prepared-txns: all temp-install
- $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
diff --git a/src/test/isolation/expected/prepared-transactions-cic.out b/src/test/isolation/expected/prepared-transactions-cic.out
new file mode 100644
index 0000000000..043ec3c363
--- /dev/null
+++ b/src/test/isolation/expected/prepared-transactions-cic.out
@@ -0,0 +1,18 @@
+Parsed test spec with 2 sessions
+
+starting permutation: w1 p1 cic2 c1 r2
+step w1: BEGIN; INSERT INTO cic_test VALUES (1);
+step p1: PREPARE TRANSACTION 's1';
+step cic2:
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+
+ERROR: canceling statement due to lock timeout
+step c1: COMMIT PREPARED 's1';
+step r2:
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+
+a
+
+1
diff --git a/src/test/isolation/specs/prepared-transactions-cic.spec b/src/test/isolation/specs/prepared-transactions-cic.spec
new file mode 100644
index 0000000000..65e2c00b4c
--- /dev/null
+++ b/src/test/isolation/specs/prepared-transactions-cic.spec
@@ -0,0 +1,34 @@
+# This test verifies that CREATE INDEX CONCURRENTLY interact with
+# prepared transactions correctly.
+setup
+{
+ CREATE TABLE cic_test (a int);
+}
+
+teardown
+{
+ DROP TABLE cic_test;
+}
+
+
+# Sessions for CREATE INDEX CONCURRENTLY test
+session "s1"
+step "w1" { BEGIN; INSERT INTO cic_test VALUES (1); }
+step "p1" { PREPARE TRANSACTION 's1'; }
+step "c1" { COMMIT PREPARED 's1'; }
+
+session "s2"
+setup { SET lock_timeout = 10; }
+step "cic2"
+{
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+}
+step "r2"
+{
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+}
+
+
+permutation "w1" "p1" "cic2" "c1" "r2"
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-01-22 05:49 ` Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 107+ messages in thread
From: Andrey Borodin @ 2021-01-22 05:49 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 22 янв. 2021 г., в 10:44, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> <v5-0001-Wait-for-prepared-xacts-in-CREATE-INDEX-CONCURREN.patch><v5-0002-Add-test-for-CIC-with-prepared-xacts.patch>
Uh, vscode did not save files and I've send incorrect version. Disregard v5.
Sorry for the noise.
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
v6-0002-Add-test-for-CIC-with-prepared-xacts.patch (application/octet-stream, 3.1K) download
From 249671aaaddee54670f48309a137bed3524dec8c Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 20 Dec 2020 23:03:01 +0500
Subject: [PATCH v6 2/2] Add test for CIC with prepared xacts
---
src/test/isolation/Makefile | 4 +--
.../expected/prepared-transactions-cic.out | 18 ++++++++++
.../specs/prepared-transactions-cic.spec | 34 +++++++++++++++++++
3 files changed, 54 insertions(+), 2 deletions(-)
create mode 100644 src/test/isolation/expected/prepared-transactions-cic.out
create mode 100644 src/test/isolation/specs/prepared-transactions-cic.spec
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index d23e2cec64..a1c3248eb4 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -67,7 +67,7 @@ check: all
# via TEMP_CONFIG for the check case, or via the postgresql.conf for the
# installcheck case.
installcheck-prepared-txns: all temp-install
- $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
check-prepared-txns: all temp-install
- $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
diff --git a/src/test/isolation/expected/prepared-transactions-cic.out b/src/test/isolation/expected/prepared-transactions-cic.out
new file mode 100644
index 0000000000..043ec3c363
--- /dev/null
+++ b/src/test/isolation/expected/prepared-transactions-cic.out
@@ -0,0 +1,18 @@
+Parsed test spec with 2 sessions
+
+starting permutation: w1 p1 cic2 c1 r2
+step w1: BEGIN; INSERT INTO cic_test VALUES (1);
+step p1: PREPARE TRANSACTION 's1';
+step cic2:
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+
+ERROR: canceling statement due to lock timeout
+step c1: COMMIT PREPARED 's1';
+step r2:
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+
+a
+
+1
diff --git a/src/test/isolation/specs/prepared-transactions-cic.spec b/src/test/isolation/specs/prepared-transactions-cic.spec
new file mode 100644
index 0000000000..65e2c00b4c
--- /dev/null
+++ b/src/test/isolation/specs/prepared-transactions-cic.spec
@@ -0,0 +1,34 @@
+# This test verifies that CREATE INDEX CONCURRENTLY interact with
+# prepared transactions correctly.
+setup
+{
+ CREATE TABLE cic_test (a int);
+}
+
+teardown
+{
+ DROP TABLE cic_test;
+}
+
+
+# Sessions for CREATE INDEX CONCURRENTLY test
+session "s1"
+step "w1" { BEGIN; INSERT INTO cic_test VALUES (1); }
+step "p1" { PREPARE TRANSACTION 's1'; }
+step "c1" { COMMIT PREPARED 's1'; }
+
+session "s2"
+setup { SET lock_timeout = 10; }
+step "cic2"
+{
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+}
+step "r2"
+{
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+}
+
+
+permutation "w1" "p1" "cic2" "c1" "r2"
--
2.24.3 (Apple Git-128)
=
v6-0001-Wait-for-prepared-xacts-in-CREATE-INDEX-CONCURREN.patch (application/octet-stream, 4.9K) download
From 27eea0d2b76808bd3f5f307ac309bad3264d8844 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 19 Dec 2020 21:53:00 +0500
Subject: [PATCH v6 1/2] Wait for prepared xacts in CREATE INDEX CONCURRENTLY
locks infrastructure
---
src/backend/storage/lmgr/lmgr.c | 3 +--
src/backend/storage/lmgr/lock.c | 31 +++++++++++++++++++++----------
src/include/storage/lock.h | 5 +++--
3 files changed, 25 insertions(+), 14 deletions(-)
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 7407672c9d..5736d03083 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -904,8 +904,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
- * check for that. Also, prepared xacts are not reported, which is fine
- * since they certainly aren't going to do anything anymore.
+ * check for that. Also, prepared xacts are reported and awaited.
*/
/* Finally wait for each such transaction to complete */
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 20e50247ea..d8a1b4ad47 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2903,9 +2903,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
* so use of this function has to be thought about carefully.
*
* Note we never include the current xact's vxid in the result array,
- * since an xact never blocks itself. Also, prepared transactions are
- * ignored, which is a bit more debatable but is appropriate for current
- * uses of the result.
+ * since an xact never blocks itself.
*/
VirtualTransactionId *
GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
@@ -2930,19 +2928,21 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* Allocate memory to store results, and fill with InvalidVXID. We only
- * need enough space for MaxBackends + a terminator, since prepared xacts
- * don't count. InHotStandby allocate once in TopMemoryContext.
+ * need enough space for MaxBackends + max_prepared_xacts + a
+ * terminator. InHotStandby allocate once in TopMemoryContext.
*/
if (InHotStandby)
{
if (vxids == NULL)
vxids = (VirtualTransactionId *)
MemoryContextAlloc(TopMemoryContext,
- sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ sizeof(VirtualTransactionId) * (MaxBackends +
+ max_prepared_xacts + 1));
}
else
vxids = (VirtualTransactionId *)
- palloc0(sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ palloc0(sizeof(VirtualTransactionId) * (MaxBackends +
+ max_prepared_xacts + 1));
/* Compute hash code and partition lock, and look up conflicting modes. */
hashcode = LockTagHashCode(locktag);
@@ -3084,8 +3084,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* If we see an invalid VXID, then either the xact has already
- * committed (or aborted), or it's a prepared xact. In either
- * case we may ignore it.
+ * committed (or aborted), or it's a prepared xact. We take
+ * prepared xacts into account too.
*/
if (VirtualTransactionIdIsValid(vxid))
{
@@ -3107,7 +3107,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
LWLockRelease(partitionLock);
- if (count > MaxBackends) /* should never happen */
+ if (count > MaxBackends + max_prepared_xacts) /* should never happen */
elog(PANIC, "too many conflicting locks found");
vxids[count].backendId = InvalidBackendId;
@@ -4464,6 +4464,17 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
Assert(VirtualTransactionIdIsValid(vxid));
+ if (VirtualTransactionIdIsPreparedXact(vxid))
+ {
+ LockAcquireResult lar;
+ /* If it's prepared xact - just wait for it */
+ SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_OK)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
+
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
/*
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 2e6ef174e9..221f23e5a6 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -68,8 +68,9 @@ typedef struct
#define InvalidLocalTransactionId 0
#define LocalTransactionIdIsValid(lxid) ((lxid) != InvalidLocalTransactionId)
#define VirtualTransactionIdIsValid(vxid) \
- (((vxid).backendId != InvalidBackendId) && \
- LocalTransactionIdIsValid((vxid).localTransactionId))
+ (LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsPreparedXact(vxid) \
+ ((vxid).backendId == InvalidBackendId)
#define VirtualTransactionIdEquals(vxid1, vxid2) \
((vxid1).backendId == (vxid2).backendId && \
(vxid1).localTransactionId == (vxid2).localTransactionId)
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-01-24 02:27 ` Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-01-24 02:27 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Fri, Jan 22, 2021 at 10:44:35AM +0500, Andrey Borodin wrote:
> > 17 янв. 2021 г., в 12:24, Noah Misch <noah@leadboat.com> написал(а):
> >> @@ -4461,9 +4462,21 @@ bool
> >> VirtualXactLock(VirtualTransactionId vxid, bool wait)
> >> {
> >> LOCKTAG tag;
> >> - PGPROC *proc;
> >> + PGPROC *proc = NULL;
> >>
> >> - Assert(VirtualTransactionIdIsValid(vxid));
> >> + Assert(VirtualTransactionIdIsValid(vxid) ||
> >> + VirtualTransactionIdIsPreparedXact(vxid));
> >> +
> >> + if (VirtualTransactionIdIsPreparedXact(vxid))
> >> + {
> >> + LockAcquireResult lar;
> >> + /* If it's prepared xact - just wait for it */
> >> + SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
> >> + lar = LockAcquire(&tag, ShareLock, false, !wait);
> >> + if (lar == LOCKACQUIRE_OK)
> >
> > This should instead test "!= LOCKACQUIRE_NOT_AVAIL", lest
> > LOCKACQUIRE_ALREADY_HELD happen. (It perhaps can't happen, but skipping the
> > LockRelease() would be wrong if it did.)
> I think that code that successfully acquired lock should release it. Other way we risk to release someone's else lock held for a reason. We only acquire lock to release it instantly anyway.
LockAcquire() increments the reference count before returning
LOCKACQUIRE_ALREADY_HELD, so it is an acquire. If this caller doesn't
LockRelease(), the lock will persist until end of transaction.
I changed that, updated comments, and fixed pgindent damage. I plan to push
the attached version.
> > The array can contain many of these new "invalid"
> > VXIDs, and an all-zeroes VXID terminates the array. Rather than change the
Correction: the terminator contains (InvalidBackendId,0).
> By the way maybe rename "check-prepared-txns" to "check-prepared-xacts" for consistency?
The "xact" term is about three times as frequent "txn". I favor "xact" in new
usage. It's not dominant enough to change old usage unless done pervasively.
Attachments:
prepared-transactions-cic-v7nm.patch (text/plain, 11.0K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
Fix CREATE INDEX CONCURRENTLY for simultaneous prepared transactions.
In a cluster having used CREATE INDEX CONCURRENTLY while having enabled
prepared transactions, queries that use the resulting index can silently
fail to find rows. Fix this for future CREATE INDEX CONCURRENTLY by
making it wait for prepared transactions like it waits for ordinary
transactions. This expands the VirtualTransactionId structure domain to
admit prepared transactions. It may be necessary to reindex to recover
from past occurrences. Back-patch to 9.5 (all supported versions).
Andrey Borodin, reviewed (in earlier versions) by Tom Lane and Michael
Paquier.
Discussion: https://postgr.es/m/2E712143-97F7-4890-B470-4A35142ABC82@yandex-team.ru
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 7407672..5736d03 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -904,8 +904,7 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
- * check for that. Also, prepared xacts are not reported, which is fine
- * since they certainly aren't going to do anything anymore.
+ * check for that. Also, prepared xacts are reported and awaited.
*/
/* Finally wait for each such transaction to complete */
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 20e5024..79c1cf9 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2903,9 +2903,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
* so use of this function has to be thought about carefully.
*
* Note we never include the current xact's vxid in the result array,
- * since an xact never blocks itself. Also, prepared transactions are
- * ignored, which is a bit more debatable but is appropriate for current
- * uses of the result.
+ * since an xact never blocks itself.
*/
VirtualTransactionId *
GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
@@ -2930,19 +2928,21 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* Allocate memory to store results, and fill with InvalidVXID. We only
- * need enough space for MaxBackends + a terminator, since prepared xacts
- * don't count. InHotStandby allocate once in TopMemoryContext.
+ * need enough space for MaxBackends + max_prepared_xacts + a terminator.
+ * InHotStandby allocate once in TopMemoryContext.
*/
if (InHotStandby)
{
if (vxids == NULL)
vxids = (VirtualTransactionId *)
MemoryContextAlloc(TopMemoryContext,
- sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ sizeof(VirtualTransactionId) *
+ (MaxBackends + max_prepared_xacts + 1));
}
else
vxids = (VirtualTransactionId *)
- palloc0(sizeof(VirtualTransactionId) * (MaxBackends + 1));
+ palloc0(sizeof(VirtualTransactionId) *
+ (MaxBackends + max_prepared_xacts + 1));
/* Compute hash code and partition lock, and look up conflicting modes. */
hashcode = LockTagHashCode(locktag);
@@ -3017,13 +3017,9 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
- /*
- * If we see an invalid VXID, then either the xact has already
- * committed (or aborted), or it's a prepared xact. In either
- * case we may ignore it.
- */
if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
+ /* else, xact already committed or aborted */
/* No need to examine remaining slots. */
break;
@@ -3082,11 +3078,6 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
- /*
- * If we see an invalid VXID, then either the xact has already
- * committed (or aborted), or it's a prepared xact. In either
- * case we may ignore it.
- */
if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -3098,6 +3089,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
if (i >= fast_count)
vxids[count++] = vxid;
}
+ /* else, xact already committed or aborted */
}
}
@@ -3107,7 +3099,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
LWLockRelease(partitionLock);
- if (count > MaxBackends) /* should never happen */
+ if (count > MaxBackends + max_prepared_xacts) /* should never happen */
elog(PANIC, "too many conflicting locks found");
vxids[count].backendId = InvalidBackendId;
@@ -4464,6 +4456,21 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
Assert(VirtualTransactionIdIsValid(vxid));
+ if (VirtualTransactionIdIsPreparedXact(vxid))
+ {
+ LockAcquireResult lar;
+
+ /*
+ * Prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
+ SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar != LOCKACQUIRE_NOT_AVAIL)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
+
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
/*
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 2e6ef17..68a3487 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -46,10 +46,10 @@ extern bool Debug_deadlocks;
/*
* Top-level transactions are identified by VirtualTransactionIDs comprising
- * the BackendId of the backend running the xact, plus a locally-assigned
- * LocalTransactionId. These are guaranteed unique over the short term,
- * but will be reused after a database restart; hence they should never
- * be stored on disk.
+ * PGPROC fields backendId and lxid. For prepared transactions, the
+ * LocalTransactionId is an ordinary XID. These are guaranteed unique over
+ * the short term, but will be reused after a database restart or XID
+ * wraparound; hence they should never be stored on disk.
*
* Note that struct VirtualTransactionId can not be assumed to be atomically
* assignable as a whole. However, type LocalTransactionId is assumed to
@@ -61,15 +61,16 @@ extern bool Debug_deadlocks;
*/
typedef struct
{
- BackendId backendId; /* determined at backend startup */
- LocalTransactionId localTransactionId; /* backend-local transaction id */
+ BackendId backendId; /* backendId from PGPROC */
+ LocalTransactionId localTransactionId; /* lxid from PGPROC */
} VirtualTransactionId;
#define InvalidLocalTransactionId 0
#define LocalTransactionIdIsValid(lxid) ((lxid) != InvalidLocalTransactionId)
#define VirtualTransactionIdIsValid(vxid) \
- (((vxid).backendId != InvalidBackendId) && \
- LocalTransactionIdIsValid((vxid).localTransactionId))
+ (LocalTransactionIdIsValid((vxid).localTransactionId))
+#define VirtualTransactionIdIsPreparedXact(vxid) \
+ ((vxid).backendId == InvalidBackendId)
#define VirtualTransactionIdEquals(vxid1, vxid2) \
((vxid1).backendId == (vxid2).backendId && \
(vxid1).localTransactionId == (vxid2).localTransactionId)
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index d23e2ce..edff04f 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -62,12 +62,11 @@ installcheck: all
check: all
$(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule
-# Versions of the check tests that include the prepared-transactions test
-# It only makes sense to run these if set up to use prepared transactions,
-# via TEMP_CONFIG for the check case, or via the postgresql.conf for the
-# installcheck case.
+# Non-default tests. It only makes sense to run these if set up to use
+# prepared transactions, via TEMP_CONFIG for the check case, or via the
+# postgresql.conf for the installcheck case.
installcheck-prepared-txns: all temp-install
- $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
check-prepared-txns: all temp-install
- $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions
+ $(pg_isolation_regress_check) --schedule=$(srcdir)/isolation_schedule prepared-transactions prepared-transactions-cic
diff --git a/src/test/isolation/README b/src/test/isolation/README
index 217953d..6ae7152 100644
--- a/src/test/isolation/README
+++ b/src/test/isolation/README
@@ -23,9 +23,9 @@ you can do something like
./pg_isolation_regress fk-contention fk-deadlock
(look into the specs/ subdirectory to see the available tests).
-The prepared-transactions test requires the server's
-max_prepared_transactions parameter to be set to at least 3; therefore it
-is not run by default. To include it in the test run, use
+Certain tests require the server's max_prepared_transactions parameter to be
+set to at least 3; therefore they are not run by default. To include them in
+the test run, use
make check-prepared-txns
or
make installcheck-prepared-txns
diff --git a/src/test/isolation/expected/prepared-transactions-cic.out b/src/test/isolation/expected/prepared-transactions-cic.out
new file mode 100644
index 0000000..043ec3c
--- /dev/null
+++ b/src/test/isolation/expected/prepared-transactions-cic.out
@@ -0,0 +1,18 @@
+Parsed test spec with 2 sessions
+
+starting permutation: w1 p1 cic2 c1 r2
+step w1: BEGIN; INSERT INTO cic_test VALUES (1);
+step p1: PREPARE TRANSACTION 's1';
+step cic2:
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+
+ERROR: canceling statement due to lock timeout
+step c1: COMMIT PREPARED 's1';
+step r2:
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+
+a
+
+1
diff --git a/src/test/isolation/specs/prepared-transactions-cic.spec b/src/test/isolation/specs/prepared-transactions-cic.spec
new file mode 100644
index 0000000..c39eaf5
--- /dev/null
+++ b/src/test/isolation/specs/prepared-transactions-cic.spec
@@ -0,0 +1,37 @@
+# This test verifies that CREATE INDEX CONCURRENTLY interacts with prepared
+# transactions correctly.
+setup
+{
+ CREATE TABLE cic_test (a int);
+}
+
+teardown
+{
+ DROP TABLE cic_test;
+}
+
+
+# Sessions for CREATE INDEX CONCURRENTLY test
+session "s1"
+step "w1" { BEGIN; INSERT INTO cic_test VALUES (1); }
+step "p1" { PREPARE TRANSACTION 's1'; }
+step "c1" { COMMIT PREPARED 's1'; }
+
+session "s2"
+# The isolation tester never recognizes that a lock of s1 blocks s2, because a
+# prepared transaction's locks have no pid associated. While there's a slight
+# chance of timeout while waiting for an autovacuum-held lock, that wouldn't
+# change the output. Hence, no timeout is too short.
+setup { SET lock_timeout = 10; }
+step "cic2"
+{
+ CREATE INDEX CONCURRENTLY on cic_test(a);
+}
+step "r2"
+{
+ SET enable_seqscan to off;
+ SET enable_bitmapscan to off;
+ SELECT * FROM cic_test WHERE a = 1;
+}
+
+permutation "w1" "p1" "cic2" "c1" "r2"
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-01-30 16:06 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-01-30 16:06 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 24 янв. 2021 г., в 07:27, Noah Misch <noah@leadboat.com> написал(а):
>
> I changed that, updated comments, and fixed pgindent damage. I plan to push
> the attached version.
I see that patch was pushed. I'll flip status of CF entry. Many thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-05-01 12:42 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-05-01 12:42 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 30 янв. 2021 г., в 21:06, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
>
>
>> 24 янв. 2021 г., в 07:27, Noah Misch <noah@leadboat.com> написал(а):
>>
>> I changed that, updated comments, and fixed pgindent damage. I plan to push
>> the attached version.
>
> I see that patch was pushed. I'll flip status of CF entry. Many thanks!
FWIW I have 2 new reported cases on 12.6. I've double-checked that at the moment of corruption installation run version with the patch.
To the date I could not reproduce the problem myself, but I'll continue working on this.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-17 17:01 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-17 17:01 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 1 мая 2021 г., в 17:42, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
>
> FWIW I have 2 new reported cases on 12.6.
Sorry to say that, but $subj persists. Here's a simple reproduction.
To get corrupted index you need 3 psql sessions A, B and C. By default command is executed in A.
create extension amcheck ;
create table t1(i int);
create index on t1(i);
begin;
insert into t1 values(0);
-- session C: reindex table concurrently t1;
prepare transaction 'a';
begin;
insert into t1 values(0);
-- session B: commit prepared 'a';
prepare transaction 'b';
begin;
insert into t1 values(0);
-- session B: commit prepared 'b';
prepare transaction 'c';
begin;
insert into t1 values(0);
-- session B: commit prepared 'c';
prepare transaction 'd';
commit prepared 'd';
-- session C: postgres=# select bt_index_check('i1',true);
ERROR: heap tuple (0,2) from table "t1" lacks matching index tuple within index "i1"
HINT: Retrying verification using the function bt_index_parent_check() might provide a more specific error.
The problem is WaitForLockersMultiple() gathers running vxids and 2pc xids. Then it waits, but if vxid is converted to 2pc it fails to wait.
I could not compose isolation test for this, because we need to do "prepare transaction 'a';" only when "reindex table concurrently t1;" is already working for some time.
To fix it we can return locking xids along with vxids from GetLockConflicts() like in attached diff. But this approach seems ugly.
Best regards, Andrey Borodin.
Attachments:
return_xids_with_vxids.diff (application/octet-stream, 2.0K) download
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..fef6ad37a5 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2928,7 +2928,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/*
* Allocate memory to store results, and fill with InvalidVXID. We only
- * need enough space for MaxBackends + max_prepared_xacts + a terminator.
+ * need enough space for MaxBackends * 2 + max_prepared_xacts + a terminator.
* InHotStandby allocate once in TopMemoryContext.
*/
if (InHotStandby)
@@ -2937,12 +2937,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
vxids = (VirtualTransactionId *)
MemoryContextAlloc(TopMemoryContext,
sizeof(VirtualTransactionId) *
- (MaxBackends + max_prepared_xacts + 1));
+ (MaxBackends * 2 + max_prepared_xacts + 1));
}
else
vxids = (VirtualTransactionId *)
palloc0(sizeof(VirtualTransactionId) *
- (MaxBackends + max_prepared_xacts + 1));
+ (MaxBackends * 2 + max_prepared_xacts + 1));
/* Compute hash code and partition lock, and look up conflicting modes. */
hashcode = LockTagHashCode(locktag);
@@ -3019,6 +3019,13 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
+
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
/* else, xact already committed or aborted */
/* No need to examine remaining slots. */
@@ -3089,6 +3096,13 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
if (i >= fast_count)
vxids[count++] = vxid;
}
+
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
/* else, xact already committed or aborted */
}
}
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-17 20:12 ` Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-07-17 20:12 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Sat, Jul 17, 2021 at 10:01:00PM +0500, Andrey Borodin wrote:
> Sorry to say that, but $subj persists. Here's a simple reproduction.
> The problem is WaitForLockersMultiple() gathers running vxids and 2pc xids. Then it waits, but if vxid is converted to 2pc it fails to wait.
That makes sense.
> To fix it we can return locking xids along with vxids from GetLockConflicts() like in attached diff. But this approach seems ugly.
Suppose some transaction has a vxid but no xid. Shortly after
GetLockConflicts(), it acquires an xid, modifies the table, and issues PREPARE
TRANSACTION. Could that cause a corrupt index even with this diff?
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-18 05:17 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-18 05:17 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 18 июля 2021 г., в 01:12, Noah Misch <noah@leadboat.com> написал(а):
>
> Suppose some transaction has a vxid but no xid. Shortly after
> GetLockConflicts(), it acquires an xid, modifies the table, and issues PREPARE
> TRANSACTION. Could that cause a corrupt index even with this diff?
Firstly I've tried to stress things out. This little go program [0] easily reproduces corruption on patched code.
Meanwhile vxid->xid->2px program does not [1] (both patched and unpatched).
I think CIC does not care much about VXIDs at all. It's only important when real XID started: before GetLockConflicts() or after.
Thanks!
Best regards, Andrey Borodin.
[0] https://gist.github.com/x4m/8b6025995eedf29bf588727375014dfc#file-stress-xid-2px
[1] https://gist.github.com/x4m/8b6025995eedf29bf588727375014dfc#file-stress-vxid-xid-2px
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-19 00:30 ` Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-07-19 00:30 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Sun, Jul 18, 2021 at 10:17:28AM +0500, Andrey Borodin wrote:
> > 18 июля 2021 г., в 01:12, Noah Misch <noah@leadboat.com> написал(а):
> >
> > Suppose some transaction has a vxid but no xid. Shortly after
> > GetLockConflicts(), it acquires an xid, modifies the table, and issues PREPARE
> > TRANSACTION. Could that cause a corrupt index even with this diff?
>
> Firstly I've tried to stress things out. This little go program [0] easily reproduces corruption on patched code.
By "patched code", you mean return_xids_with_vxids.diff applied to recent
master, right?
> Meanwhile vxid->xid->2px program does not [1] (both patched and unpatched).
>
> I think CIC does not care much about VXIDs at all. It's only important when real XID started: before GetLockConflicts() or after.
CIC likes VXIDs because they're assigned early in the xact, before any catalog
access. Consider the DefineIndex() step "wait until no running transaction
could have the table open with the old list of indexes." ExecOpenIndices()
can happen before AssignTransactionId(), so waiting on XIDs alone wouldn't
suffice.
To fix $SUBJECT, it sounds like we need a way to identify a transaction,
usable as early as the transaction's first catalog access and remaining valid
until COMMIT PREPARED finishes. We may initially see a transaction as having
a VXID and no XID, then later need to wait for that transaction when it has
entered prepared state, having an XID and no VXID. How might we achieve that?
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-19 07:10 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-19 07:10 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 19 июля 2021 г., в 05:30, Noah Misch <noah@leadboat.com> написал(а):
>
> To fix $SUBJECT, it sounds like we need a way to identify a transaction,
> usable as early as the transaction's first catalog access and remaining valid
> until COMMIT PREPARED finishes. We may initially see a transaction as having
> a VXID and no XID, then later need to wait for that transaction when it has
> entered prepared state, having an XID and no VXID. How might we achieve that?
PFA draft with vxid->xid mapping and subsequent wait for it. The patch, obviously, lacks a ton of comments explaining what is going on.
We write actual VXID into dummy proc entries of prepared xact.
When we wait for vxid we try to convert it to xid through real proc entry. If we cannot do so - we lookup in shared 2pc state. If vxid is not there - it means it is already gone and there's nothing to wait.
Thanks!
Best regards, Andrey Borodin.
Attachments:
v2-0001-Wait-for-vxids-in-CIC.patch (application/octet-stream, 6.6K) download
From 8739848c4ade4616f88c0ac93cf082554c8f1e13 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v2] Wait for vxids in CIC
---
src/backend/access/transam/twophase.c | 41 +++++++++++++--
src/backend/storage/lmgr/lock.c | 72 ++++++++++++++++++++-------
src/include/access/twophase.h | 2 +
3 files changed, 94 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..dba302d322 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -373,8 +373,10 @@ MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at, Oid owner, Oid databaseid)
{
GlobalTransaction gxact;
+
int i;
+
if (strlen(gid) >= GIDSIZE)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -459,14 +461,16 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +850,37 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+TransactionId
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ TransactionId result = InvalidTransactionId;
+ int i;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc;
+ proc = &ProcGlobal->allProcs[gxact->pgprocno];
+
+ if (proc->backendId == vxid.backendId &&
+ proc->lxid == vxid.localTransactionId)
+ {
+ result = gxact->xid;
+ break;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..25e3bb6fda 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -4447,6 +4448,45 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * PreparedXactLock
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * fake procarray entries.
+ */
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+
+ /* Questionable heuristics */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ if (!TransactionIdIsValid(xid))
+ {
+ /* We must search for vxid in 2pc state */
+ /* XXX: O(N*N) complexity where N is number of prepared xacts */
+ xid = TwoPhaseGetXidByVXid(vxid);
+ if (TransactionIdIsValid(xid)) // Remove this line
+ { // Remove this line
+ elog(NOTICE,"XXX: Sucessfully found xid by vxid"); // Remove this line
+ } // Remove this line
+ }
+
+ if (!TransactionIdIsValid(xid))
+ {
+ /* Seems like vxid's 2pc is gone or never existed */
+ return true;
+ }
+
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar != LOCKACQUIRE_NOT_AVAIL)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4499,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return PreparedXactLock(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4524,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4538,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4595,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return PreparedXactLock(vxid, xid, wait);;
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..19923555ed 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -58,4 +58,6 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+extern TransactionId TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-19 18:10 ` Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-07-19 18:10 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Mon, Jul 19, 2021 at 12:10:52PM +0500, Andrey Borodin wrote:
> > 19 июля 2021 г., в 05:30, Noah Misch <noah@leadboat.com> написал(а):
> >
> > To fix $SUBJECT, it sounds like we need a way to identify a transaction,
> > usable as early as the transaction's first catalog access and remaining valid
> > until COMMIT PREPARED finishes. We may initially see a transaction as having
> > a VXID and no XID, then later need to wait for that transaction when it has
> > entered prepared state, having an XID and no VXID. How might we achieve that?
>
> PFA draft with vxid->xid mapping and subsequent wait for it. The patch, obviously, lacks a ton of comments explaining what is going on.
> We write actual VXID into dummy proc entries of prepared xact.
> When we wait for vxid we try to convert it to xid through real proc entry. If we cannot do so - we lookup in shared 2pc state. If vxid is not there - it means it is already gone and there's nothing to wait.
When the system reuses BackendId values, it reuses VXID values. In the
general case, two prepared transactions could exist simultaneously with the
same BackendId+LocalTransactionId. Hmm. It could be okay to have a small
probability that CIC waits on more transactions than necessary. Suppose we
have three PGPROC entries with the same VXID, two prepared transactions and
one regular transaction. Waiting for all three could be tolerable, though
avoiding that would be nice. Should we follow transactions differently to
avoid that?
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-19 18:41 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 107+ messages in thread
From: Andrey Borodin @ 2021-07-19 18:41 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 19 июля 2021 г., в 23:10, Noah Misch <noah@leadboat.com> написал(а):
>
> On Mon, Jul 19, 2021 at 12:10:52PM +0500, Andrey Borodin wrote:
>>>> 19 июля 2021 г., в 05:30, Noah Misch <noah@leadboat.com> написал(а):
>>>
>>> To fix $SUBJECT, it sounds like we need a way to identify a transaction,
>>> usable as early as the transaction's first catalog access and remaining valid
>>> until COMMIT PREPARED finishes. We may initially see a transaction as having
>>> a VXID and no XID, then later need to wait for that transaction when it has
>>> entered prepared state, having an XID and no VXID. How might we achieve that?
>>
>> PFA draft with vxid->xid mapping and subsequent wait for it. The patch, obviously, lacks a ton of comments explaining what is going on.
>> We write actual VXID into dummy proc entries of prepared xact.
>> When we wait for vxid we try to convert it to xid through real proc entry. If we cannot do so - we lookup in shared 2pc state. If vxid is not there - it means it is already gone and there's nothing to wait.
>
> When the system reuses BackendId values, it reuses VXID values. In the
> general case, two prepared transactions could exist simultaneously with the
> same BackendId+LocalTransactionId. Hmm. It could be okay to have a small
> probability that CIC waits on more transactions than necessary. Suppose we
> have three PGPROC entries with the same VXID, two prepared transactions and
> one regular transaction. Waiting for all three could be tolerable, though
> avoiding that would be nice. Should we follow transactions differently to
> avoid that?
We don’t have to wait for regular Xid in this case at all. Because it would be finished with VXID. But I think that we have to wait for all 2PCs with the same VXID.
We are looking for transaction that was only VXID during GetLockConflicts(). In conflicts array we may have each VXID only once.
Other 2PCs with same VXID may be older or newer than target 2PC.
Older 2PCs must be with XID in conflicts array. So we might wait for all 2PC with known XIDs. Then for each ambiguous VXID->XID mapping choose oldest XID.
But this logic seem to me overly complicated. Or isn’t it?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-20 19:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 107+ messages in thread
From: Andrey Borodin @ 2021-07-20 19:25 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 19 июля 2021 г., в 23:41, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
>
> We are looking for transaction that was only VXID during GetLockConflicts(). In conflicts array we may have each VXID only once.
> Other 2PCs with same VXID may be older or newer than target 2PC.
> Older 2PCs must be with XID in conflicts array. So we might wait for all 2PC with known XIDs. Then for each ambiguous VXID->XID mapping choose oldest XID.
>
> But this logic seem to me overly complicated. Or isn’t it?
Here's the PoC to asses complexity of this solution.
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
v3-0001-Wait-for-vxids-in-CIC.patch (application/octet-stream, 6.6K) download
From 8739848c4ade4616f88c0ac93cf082554c8f1e13 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v3 1/2] Wait for vxids in CIC
---
src/backend/access/transam/twophase.c | 41 +++++++++++++--
src/backend/storage/lmgr/lock.c | 72 ++++++++++++++++++++-------
src/include/access/twophase.h | 2 +
3 files changed, 94 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..dba302d322 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -373,8 +373,10 @@ MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at, Oid owner, Oid databaseid)
{
GlobalTransaction gxact;
+
int i;
+
if (strlen(gid) >= GIDSIZE)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -459,14 +461,16 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +850,37 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+TransactionId
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ TransactionId result = InvalidTransactionId;
+ int i;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc;
+ proc = &ProcGlobal->allProcs[gxact->pgprocno];
+
+ if (proc->backendId == vxid.backendId &&
+ proc->lxid == vxid.localTransactionId)
+ {
+ result = gxact->xid;
+ break;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..25e3bb6fda 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -4447,6 +4448,45 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * PreparedXactLock
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * fake procarray entries.
+ */
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+
+ /* Questionable heuristics */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ if (!TransactionIdIsValid(xid))
+ {
+ /* We must search for vxid in 2pc state */
+ /* XXX: O(N*N) complexity where N is number of prepared xacts */
+ xid = TwoPhaseGetXidByVXid(vxid);
+ if (TransactionIdIsValid(xid)) // Remove this line
+ { // Remove this line
+ elog(NOTICE,"XXX: Sucessfully found xid by vxid"); // Remove this line
+ } // Remove this line
+ }
+
+ if (!TransactionIdIsValid(xid))
+ {
+ /* Seems like vxid's 2pc is gone or never existed */
+ return true;
+ }
+
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar != LOCKACQUIRE_NOT_AVAIL)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4499,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return PreparedXactLock(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4524,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4538,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4595,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return PreparedXactLock(vxid, xid, wait);;
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..19923555ed 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -58,4 +58,6 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+extern TransactionId TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v3-0002-Partition-2PCs-first-during-wait.patch (application/octet-stream, 2.7K) download
From a4db2c29cb07e0129af2734a31299bae9bdfe680 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Wed, 21 Jul 2021 00:24:13 +0500
Subject: [PATCH v3 2/2] Partition 2PCs first during wait
---
src/backend/access/transam/twophase.c | 7 +++---
src/backend/storage/lmgr/lmgr.c | 32 +++++++++++++++++++++++----
2 files changed, 32 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index dba302d322..836c9339af 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -852,7 +852,7 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
/*
* TwoPhaseGetXidByVXid
- * Try to lookup for vxid among prepared xacts
+ * Try to lookup earliest xid for vxid among prepared xacts
*/
TransactionId
TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
@@ -871,8 +871,9 @@ TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
if (proc->backendId == vxid.backendId &&
proc->lxid == vxid.localTransactionId)
{
- result = gxact->xid;
- break;
+ if (result == InvalidTransactionId ||
+ TransactionIdPrecedes(gxact->xid, result))
+ result = gxact->xid;
}
}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index cdf2266d6d..d03ce465cc 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -912,20 +912,44 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
foreach(lc, holders)
{
VirtualTransactionId *lockholders = lfirst(lc);
+ int nholders = 0;
+ int i = 0, j;
- while (VirtualTransactionIdIsValid(*lockholders))
+ while (VirtualTransactionIdIsValid(lockholders[nholders]))
+ nholders++;
+ j = nholders - 1;
+
+ /* partition 2PC first */
+ while (nholders > 0)
+ {
+ while (i < nholders && VirtualTransactionIdIsPreparedXact(lockholders[i]))
+ i++;
+ while (j >= 0 && !VirtualTransactionIdIsPreparedXact(lockholders[j]))
+ j--;
+ if (i < j)
+ {
+ VirtualTransactionId tmp = lockholders[i];
+ lockholders[i] = lockholders[j];
+ lockholders[j] = tmp;
+ i++;
+ j++;
+ }
+ if (i >= j)
+ break;
+ }
+
+ for (i = 0; i < nholders; i++)
{
/* If requested, publish who we're going to wait for. */
if (progress)
{
- PGPROC *holder = BackendIdGetProc(lockholders->backendId);
+ PGPROC *holder = BackendIdGetProc(lockholders[i].backendId);
if (holder)
pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID,
holder->pid);
}
- VirtualXactLock(*lockholders, true);
- lockholders++;
+ VirtualXactLock(lockholders[i], true);
if (progress)
pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-20 19:38 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-20 19:38 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 19 июля 2021 г., в 23:41, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
>
> We are looking for transaction that was only VXID during GetLockConflicts(). In conflicts array we may have each VXID only once.
> Other 2PCs with same VXID may be older or newer than target 2PC.
> Older 2PCs must be with XID in conflicts array. So we might wait for all 2PC with known XIDs. Then for each ambiguous VXID->XID mapping choose oldest XID.
>
> But this logic seem to me overly complicated. Or isn’t it?
> Other 2PCs with same VXID may be older or newer than target 2PC. Older 2PCs must be with XID in conflicts array.
Unfortunately, this is just wrong. Older 2PC with same VXID don't have to be in conflicts array. They might be of some other unrelated 2PC working with different relations.
Sorry for the noise.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-20 21:49 ` Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-07-20 21:49 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Wed, Jul 21, 2021 at 12:38:25AM +0500, Andrey Borodin wrote:
> > 19 июля 2021 г., в 23:41, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
> > > 19 июля 2021 г., в 23:10, Noah Misch <noah@leadboat.com> написал(а):
> > > On Mon, Jul 19, 2021 at 12:10:52PM +0500, Andrey Borodin wrote:
> > >>>> 19 июля 2021 г., в 05:30, Noah Misch <noah@leadboat.com> написал(а):
> > >>>
> > >>> To fix $SUBJECT, it sounds like we need a way to identify a transaction,
> > >>> usable as early as the transaction's first catalog access and remaining valid
> > >>> until COMMIT PREPARED finishes. We may initially see a transaction as having
> > >>> a VXID and no XID, then later need to wait for that transaction when it has
> > >>> entered prepared state, having an XID and no VXID. How might we achieve that?
> > >>
> > >> PFA draft with vxid->xid mapping and subsequent wait for it. The patch, obviously, lacks a ton of comments explaining what is going on.
> > >> We write actual VXID into dummy proc entries of prepared xact.
> > >> When we wait for vxid we try to convert it to xid through real proc entry. If we cannot do so - we lookup in shared 2pc state. If vxid is not there - it means it is already gone and there's nothing to wait.
> > >
> > > When the system reuses BackendId values, it reuses VXID values. In the
> > > general case, two prepared transactions could exist simultaneously with the
> > > same BackendId+LocalTransactionId. Hmm. It could be okay to have a small
> > > probability that CIC waits on more transactions than necessary. Suppose we
> > > have three PGPROC entries with the same VXID, two prepared transactions and
> > > one regular transaction. Waiting for all three could be tolerable, though
> > > avoiding that would be nice. Should we follow transactions differently to
> > > avoid that?
> >
> > We don’t have to wait for regular Xid in this case at all. Because it would be finished with VXID.
I don't understand those two sentences. Could you write more?
> > We are looking for transaction that was only VXID during GetLockConflicts(). In conflicts array we may have each VXID only once.
> > Other 2PCs with same VXID may be older or newer than target 2PC.
> > Older 2PCs must be with XID in conflicts array. So we might wait for all 2PC with known XIDs. Then for each ambiguous VXID->XID mapping choose oldest XID.
> Unfortunately, this is just wrong. Older 2PC with same VXID don't have to be in conflicts array. They might be of some other unrelated 2PC working with different relations.
I agree. Would it work to do the following sequence in WaitForLockers()?
1. In GetLockConflicts(), record a list of conflicting XIDs. Also record a
list of conflicting LXIDs having no XID.
2. Wait on all LXIDs recorded in (1). They have either ended or converted to
prepared transactions.
3. Inner-join the present list of prepared transactions with the list of
LXIDs from (1). Record the XIDs of matched entries.
4. Wait on all XIDs recorded in (1) and (3).
While that may wait on some prepared XIDs needlessly, it can't degrade into
persistent starvation. We could reduce the chance of a needless XID wait by
remembering the list of old prepared XIDs before (1) and not adding any of
those remembered, old XIDs in (3). That last bit probably would be
unjustified complexity, but maybe not.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-21 08:13 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-21 08:13 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 21 июля 2021 г., в 02:49, Noah Misch <noah@leadboat.com> написал(а):
>
> On Wed, Jul 21, 2021 at 12:38:25AM +0500, Andrey Borodin wrote:
>>> 19 июля 2021 г., в 23:41, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>>>> 19 июля 2021 г., в 23:10, Noah Misch <noah@leadboat.com> написал(а):
>>>> Suppose we
>>>> have three PGPROC entries with the same VXID, two prepared transactions and
>>>> one regular transaction. Waiting for all three could be tolerable, though
>>>> avoiding that would be nice. Should we follow transactions differently to
>>>> avoid that?
>>>
>>> We don’t have to wait for regular Xid in this case at all. Because it would be finished with VXID.
>
> I don't understand those two sentences. Could you write more?
Suppose we have a VXID conflicting with reindexed relation lock, and have a PGPROC with regular Xid (not 2PC) for this VXID.
We do not need to return this xid from TwoPhaseGetXidByVXid() for extra wait. This situation is covered by normal vxid handling in VirtualXactLock().
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
>>> We are looking for transaction that was only VXID during GetLockConflicts(). In conflicts array we may have each VXID only once.
>>> Other 2PCs with same VXID may be older or newer than target 2PC.
>>> Older 2PCs must be with XID in conflicts array. So we might wait for all 2PC with known XIDs. Then for each ambiguous VXID->XID mapping choose oldest XID.
>
>> Unfortunately, this is just wrong. Older 2PC with same VXID don't have to be in conflicts array. They might be of some other unrelated 2PC working with different relations.
>
> I agree. Would it work to do the following sequence in WaitForLockers()?
>
> 1. In GetLockConflicts(), record a list of conflicting XIDs. Also record a
> list of conflicting LXIDs having no XID.
> 2. Wait on all LXIDs recorded in (1). They have either ended or converted to
> prepared transactions.
> 3. Inner-join the present list of prepared transactions with the list of
> LXIDs from (1). Record the XIDs of matched entries.
> 4. Wait on all XIDs recorded in (1) and (3).
>
> While that may wait on some prepared XIDs needlessly, it can't degrade into
> persistent starvation. We could reduce the chance of a needless XID wait by
> remembering the list of old prepared XIDs before (1) and not adding any of
> those remembered, old XIDs in (3). That last bit probably would be
> unjustified complexity, but maybe not.
I think this protocol is equivalent to waiting on all Xids with VXid.
I consider this protocol safe. FPA implementation.
Patch 0001 is intact version of previous patch.
There are two additions:
1. Prefer xids to vxids in GetLockConflicts()
2. Wait on all 2PCs with given VXid.
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (3 files, .tar.gz)
v4-0001-Wait-for-vxids-in-CIC.patch (application/octet-stream, 6.6K) download
From 8739848c4ade4616f88c0ac93cf082554c8f1e13 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v4 1/3] Wait for vxids in CIC
---
src/backend/access/transam/twophase.c | 41 +++++++++++++--
src/backend/storage/lmgr/lock.c | 72 ++++++++++++++++++++-------
src/include/access/twophase.h | 2 +
3 files changed, 94 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..dba302d322 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -373,8 +373,10 @@ MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at, Oid owner, Oid databaseid)
{
GlobalTransaction gxact;
+
int i;
+
if (strlen(gid) >= GIDSIZE)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -459,14 +461,16 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +850,37 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+TransactionId
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ TransactionId result = InvalidTransactionId;
+ int i;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc;
+ proc = &ProcGlobal->allProcs[gxact->pgprocno];
+
+ if (proc->backendId == vxid.backendId &&
+ proc->lxid == vxid.localTransactionId)
+ {
+ result = gxact->xid;
+ break;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..25e3bb6fda 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -4447,6 +4448,45 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * PreparedXactLock
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * fake procarray entries.
+ */
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+
+ /* Questionable heuristics */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ if (!TransactionIdIsValid(xid))
+ {
+ /* We must search for vxid in 2pc state */
+ /* XXX: O(N*N) complexity where N is number of prepared xacts */
+ xid = TwoPhaseGetXidByVXid(vxid);
+ if (TransactionIdIsValid(xid)) // Remove this line
+ { // Remove this line
+ elog(NOTICE,"XXX: Sucessfully found xid by vxid"); // Remove this line
+ } // Remove this line
+ }
+
+ if (!TransactionIdIsValid(xid))
+ {
+ /* Seems like vxid's 2pc is gone or never existed */
+ return true;
+ }
+
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar != LOCKACQUIRE_NOT_AVAIL)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4499,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return PreparedXactLock(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4524,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4538,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4595,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return PreparedXactLock(vxid, xid, wait);;
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..19923555ed 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -58,4 +58,6 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+extern TransactionId TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v4-0002-Prefer-real-Xid-over-local.patch (application/octet-stream, 1.5K) download
From b0678f07a650454a90f7974efc7f4be876f7d722 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Wed, 21 Jul 2021 10:57:18 +0500
Subject: [PATCH v4 2/3] Prefer real Xid over local
---
src/backend/storage/lmgr/lock.c | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 25e3bb6fda..50553e729c 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -3018,7 +3018,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3079,7 +3086,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
{
int i;
--
2.24.3 (Apple Git-128)
=
v4-0003-Wait-on-many-xids-returned-from-TwoPhaseGetXidByV.patch (application/octet-stream, 4.1K) download
From f11ba01cbfd0932d0c74f1bfdbfabb484bdfe943 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Wed, 21 Jul 2021 12:21:23 +0500
Subject: [PATCH v4 3/3] Wait on many xids returned from TwoPhaseGetXidByVXid()
---
src/backend/access/transam/twophase.c | 18 +++++++++----
src/backend/storage/lmgr/lock.c | 38 ++++++++++++++++-----------
src/include/access/twophase.h | 12 ++++++++-
3 files changed, 46 insertions(+), 22 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index dba302d322..45e1c3259b 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -854,11 +854,13 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
* TwoPhaseGetXidByVXid
* Try to lookup for vxid among prepared xacts
*/
-TransactionId
+XidListEntry
TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
{
- TransactionId result = InvalidTransactionId;
- int i;
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
@@ -871,8 +873,14 @@ TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
if (proc->backendId == vxid.backendId &&
proc->lxid == vxid.localTransactionId)
{
- result = gxact->xid;
- break;
+ if (result.xid != InvalidTransactionId)
+ {
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
}
}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 50553e729c..55efb33a7b 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -4472,33 +4472,39 @@ static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool
{
LockAcquireResult lar;
LOCKTAG tag;
+ XidListEntry xidlist;
/* Questionable heuristics */
if (max_prepared_xacts == 0)
return true;
- if (!TransactionIdIsValid(xid))
+ if (TransactionIdIsValid(xid))
{
- /* We must search for vxid in 2pc state */
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /* We must search for vxids in 2pc state */
/* XXX: O(N*N) complexity where N is number of prepared xacts */
- xid = TwoPhaseGetXidByVXid(vxid);
- if (TransactionIdIsValid(xid)) // Remove this line
- { // Remove this line
- elog(NOTICE,"XXX: Sucessfully found xid by vxid"); // Remove this line
- } // Remove this line
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ elog(NOTICE,"XXX: Sucessfully found xid by vxid"); // Remove this line
}
- if (!TransactionIdIsValid(xid))
+ while (true)
{
- /* Seems like vxid's 2pc is gone or never existed */
- return true;
+ if (TransactionIdIsValid(xidlist.xid))
+ {
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar != LOCKACQUIRE_NOT_AVAIL)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
+ if (xidlist.next == NULL)
+ return true;
+ xidlist = *xidlist.next;
}
-
- SET_LOCKTAG_TRANSACTION(tag, xid);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 19923555ed..70b3cb9391 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -59,5 +59,15 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
-extern TransactionId TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
+/*
+ * This struc is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-21 17:55 ` Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-07-21 17:55 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Wed, Jul 21, 2021 at 01:13:08PM +0500, Andrey Borodin wrote:
> > 21 июля 2021 г., в 02:49, Noah Misch <noah@leadboat.com> написал(а):
> > On Wed, Jul 21, 2021 at 12:38:25AM +0500, Andrey Borodin wrote:
> >>> 19 июля 2021 г., в 23:41, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
> >>>> 19 июля 2021 г., в 23:10, Noah Misch <noah@leadboat.com> написал(а):
> >>>> Suppose we
> >>>> have three PGPROC entries with the same VXID, two prepared transactions and
> >>>> one regular transaction. Waiting for all three could be tolerable, though
> >>>> avoiding that would be nice. Should we follow transactions differently to
> >>>> avoid that?
> >>>
> >>> We don’t have to wait for regular Xid in this case at all. Because it would be finished with VXID.
> >
> > I don't understand those two sentences. Could you write more?
> Suppose we have a VXID conflicting with reindexed relation lock, and have a PGPROC with regular Xid (not 2PC) for this VXID.
> We do not need to return this xid from TwoPhaseGetXidByVXid() for extra wait. This situation is covered by normal vxid handling in VirtualXactLock().
> + /* Save the xid to test if transaction coverted to 2pc later */
> + xid = proc->xid;
Got it.
> > Would it work to do the following sequence in WaitForLockers()?
> >
> > 1. In GetLockConflicts(), record a list of conflicting XIDs. Also record a
> > list of conflicting LXIDs having no XID.
> > 2. Wait on all LXIDs recorded in (1). They have either ended or converted to
> > prepared transactions.
> > 3. Inner-join the present list of prepared transactions with the list of
> > LXIDs from (1). Record the XIDs of matched entries.
> > 4. Wait on all XIDs recorded in (1) and (3).
> >
> > While that may wait on some prepared XIDs needlessly, it can't degrade into
> > persistent starvation. We could reduce the chance of a needless XID wait by
> > remembering the list of old prepared XIDs before (1) and not adding any of
> > those remembered, old XIDs in (3). That last bit probably would be
> > unjustified complexity, but maybe not.
> I think this protocol is equivalent to waiting on all Xids with VXid.
> I consider this protocol safe. FPA implementation.
> Patch 0001 is intact version of previous patch.
> There are two additions:
> 1. Prefer xids to vxids in GetLockConflicts()
> 2. Wait on all 2PCs with given VXid.
These drafts use reasonable concepts. Would you develop them into a
ready-to-commit patch? You may be able to translate your probabilistic test
procedures from https://gist.github.com/x4m/8b6025995eedf29bf588727375014dfc
into something like the src/bin/pgbench/t/001_pgbench_with_server.pl test of
pg_enum_oid_index.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-23 16:33 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-23 16:33 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 21 июля 2021 г., в 22:55, Noah Misch <noah@leadboat.com> написал(а):
>
> These drafts use reasonable concepts. Would you develop them into a
> ready-to-commit patch? You may be able to translate your probabilistic test
> procedures from https://gist.github.com/x4m/8b6025995eedf29bf588727375014dfc
> into something like the src/bin/pgbench/t/001_pgbench_with_server.pl test of
> pg_enum_oid_index.
I'm working on it. Is it OK to use amcheck in pgbench tests? Or is it better to add the test to amcheck?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-23 22:30 ` Noah Misch <noah@leadboat.com>
2021-07-23 22:53 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 107+ messages in thread
From: Noah Misch @ 2021-07-23 22:30 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Fri, Jul 23, 2021 at 09:33:18PM +0500, Andrey Borodin wrote:
> > 21 июля 2021 г., в 22:55, Noah Misch <noah@leadboat.com> написал(а):
> > These drafts use reasonable concepts. Would you develop them into a
> > ready-to-commit patch? You may be able to translate your probabilistic test
> > procedures from https://gist.github.com/x4m/8b6025995eedf29bf588727375014dfc
> > into something like the src/bin/pgbench/t/001_pgbench_with_server.pl test of
> > pg_enum_oid_index.
>
> I'm working on it. Is it OK to use amcheck in pgbench tests? Or is it better to add the test to amcheck?
It could be okay, but I think it's better to add the test under amcheck. You
could still use pgbench in the test.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-23 22:53 ` Peter Geoghegan <pg@bowt.ie>
1 sibling, 0 replies; 107+ messages in thread
From: Peter Geoghegan @ 2021-07-23 22:53 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Jul 23, 2021 at 3:30 PM Noah Misch <noah@leadboat.com> wrote:
> It could be okay, but I think it's better to add the test under amcheck. You
> could still use pgbench in the test.
+1. Writing the test that way seems more robust. If somebody writes a
patch that reintroduces a similar though distinct bug, then it seems
more likely to be caught. In other words, there is less of a danger of
the test over-specifying what incorrect means if it uses amcheck
directly.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-27 12:50 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-27 12:50 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 24 июля 2021 г., в 03:30, Noah Misch <noah@leadboat.com> написал(а):
>
> It could be okay, but I think it's better to add the test under amcheck. You
> could still use pgbench in the test.
Currently it's still WIP.
I've added two tests: deterministic with psql and probabilistic with pgbench.
And I really do not like pgbench test:
1. It does not seem stable enough, it can turn buildfarm red as a good watermelon.
2. Names for 2PCs are choosen at random and have probability of collision.
3. It still breaks the fix and I have no idea why.
Can you please take a look on added TAP test? Probably I'm doing wrong a lot of things, it's the very first program on Perl written by me...
background_pgbench is a local invention. sub pgbench is a copy from nearby test. Should I refactor it somewhere?
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
v5-0001-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 7.0K) download
From 6edccb3906472e88b57f7642d496c42078fddc81 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v5 1/2] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/t/002_cic_2pc.pl | 238 +++++++++++++++++++++++++++++++
2 files changed, 239 insertions(+), 1 deletion(-)
create mode 100644 contrib/amcheck/t/002_cic_2pc.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..5b5427d193 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -12,7 +12,7 @@ PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
-TAP_TESTS = 1
+TAP_TESTS = 2
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/amcheck/t/002_cic_2pc.pl b/contrib/amcheck/t/002_cic_2pc.pl
new file mode 100644
index 0000000000..541ac2e8a9
--- /dev/null
+++ b/contrib/amcheck/t/002_cic_2pc.pl
@@ -0,0 +1,238 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for point-in-time-recovery (PITR) with prepared transactions
+use strict;
+use warnings;
+
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = get_new_node('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(5);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(5);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+$main_h->pump_nb;
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = background_pgbench('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --time=1');
+
+# Run pgbench.
+pgbench(
+ '--no-vacuum --client=5 --time=1',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ \set txid random(1, 1000000000)
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'tx:txid';
+ COMMIT PREPARED 'tx:txid';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = $pgbench_h->result(0); # Need extra check for ($Config{osname} eq "MSWin32") ?
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
+
+# invoke pgbench, with parameters:
+# $opts: options as a string to be split on spaces
+# $stat: expected exit status
+# $out: reference to a regexp list that must match stdout
+# $err: reference to a regexp list that must match stderr
+# $name: name of test for error messages
+# $files: reference to filename/contents dictionary
+# @args: further raw options or arguments
+sub pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my ($opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $node->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+
+# invoke pgbench in background alike background_psql
+sub background_pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my ($dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $node->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v5-0002-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxid.patch (application/octet-stream, 8.2K) download
From 515f3cd1081880b72a230487a2a9002a5e56fcdc Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v5 2/2] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2pc
---
src/backend/access/transam/twophase.c | 49 +++++++++++++-
src/backend/storage/lmgr/lock.c | 96 +++++++++++++++++++++------
src/include/access/twophase.h | 12 ++++
3 files changed, 134 insertions(+), 23 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..45e1c3259b 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -373,8 +373,10 @@ MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at, Oid owner, Oid databaseid)
{
GlobalTransaction gxact;
+
int i;
+
if (strlen(gid) >= GIDSIZE)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -459,14 +461,16 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +850,45 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+XidListEntry
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc;
+ proc = &ProcGlobal->allProcs[gxact->pgprocno];
+
+ if (proc->backendId == vxid.backendId &&
+ proc->lxid == vxid.localTransactionId)
+ {
+ if (result.xid != InvalidTransactionId)
+ {
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..55efb33a7b 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -3017,7 +3018,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3078,7 +3086,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -4447,6 +4462,51 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * PreparedXactLock
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * fake procarray entries.
+ */
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ XidListEntry xidlist;
+
+ /* Questionable heuristics */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ if (TransactionIdIsValid(xid))
+ {
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /* We must search for vxids in 2pc state */
+ /* XXX: O(N*N) complexity where N is number of prepared xacts */
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ elog(NOTICE,"XXX: Sucessfully found xid by vxid"); // Remove this line
+ }
+
+ while (true)
+ {
+ if (TransactionIdIsValid(xidlist.xid))
+ {
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar != LOCKACQUIRE_NOT_AVAIL)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
+ if (xidlist.next == NULL)
+ return true;
+ xidlist = *xidlist.next;
+ }
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4519,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return PreparedXactLock(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4544,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4558,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4615,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return PreparedXactLock(vxid, xid, wait);;
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..70b3cb9391 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -58,4 +58,16 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+/*
+ * This struc is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-29 12:30 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-29 12:30 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 27 июля 2021 г., в 17:50, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
>
> 3. It still breaks the fix and I have no idea why.
I'm still coping with the bug. I have a stable reproduction, but to the date unable to identify roots of the problem.
Here's the sample trace:
2021-07-29 17:14:28.996 +05 [61148] 002_cic_2pc.pl LOG: statement: REINDEX INDEX CONCURRENTLY idx;
2021-07-29 17:14:28.996 +05 [61148] 002_cic_2pc.pl WARNING: Phase 1
2021-07-29 17:14:28.997 +05 [61148] 002_cic_2pc.pl WARNING: Phase 2
2021-07-29 17:14:28.997 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/3493
2021-07-29 17:14:28.997 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 3493, vxid -1/3493
2021-07-29 17:14:28.997 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/3490
2021-07-29 17:14:28.997 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 3490, vxid -1/3490
2021-07-29 17:14:28.997 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/3492
2021-07-29 17:14:28.997 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 3492, vxid -1/3492
2021-07-29 17:14:28.999 +05 [61148] 002_cic_2pc.pl WARNING: Phase 3
2021-07-29 17:14:28.999 +05 [61148] 002_cic_2pc.pl WARNING: Phase 4
2021-07-29 17:14:29.000 +05 [61148] 002_cic_2pc.pl WARNING: Phase 5
2021-07-29 17:14:29.005 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/3512
2021-07-29 17:14:29.005 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 3512, vxid -1/3512
2021-07-29 17:14:29.006 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/3513
2021-07-29 17:14:29.006 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 3513, vxid -1/3513
2021-07-29 17:14:29.006 +05 [61148] 002_cic_2pc.pl WARNING: Phase 6
2021-07-29 17:14:29.006 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/3516
2021-07-29 17:14:29.006 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 3516, vxid -1/3516
2021-07-29 17:14:29.006 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/3515
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 3515, vxid -1/3515
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/3517
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 3517, vxid -1/3517
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid 4/1002
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl WARNING: Backend is doing something else
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl WARNING: XXX: PreparedXactLock xid 0, vxid 4/1002
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl WARNING: XXX: Sucessfully found xid by vxid 0
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl WARNING: Phase Final
2021-07-29 17:14:29.007 +05 [61148] 002_cic_2pc.pl LOG: statement: SELECT bt_index_check('idx',true);
2021-07-29 17:14:29.009 +05 [61148] 002_cic_2pc.pl ERROR: heap tuple (18,74) from table "tbl" lacks matching index tuple within index "idx" xmin 3504 xmax 0
2021-07-29 17:14:29.009 +05 [61148] 002_cic_2pc.pl STATEMENT: SELECT bt_index_check('idx',true);
Rogue tuple was committed by xid 3504 which was never returned by GetLockConflicts(). I'm attaching patch set just for the reference of the trace, not expecting code review now.
I've fixed unrelated bug in previous patchset.
- SET_LOCKTAG_TRANSACTION(tag, xid);
+ SET_LOCKTAG_TRANSACTION(tag, xidlist.xid);
lar = LockAcquire(&tag, ShareLock, false, !wait);
if (lar != LOCKACQUIRE_NOT_AVAIL)
LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ return false;
But it does not help. Maybe I've broken something by my fix...Searching further.
Thanks for reading! Would be happy to hear any ideas why xid was not locked by GetLockConflicts() bug committed a tuple which was not indexed.
Best regards, Andrey Borodin.
Attachments: download all (3 files, .tar.gz)
v6-0001-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 7.0K) download
From 6edccb3906472e88b57f7642d496c42078fddc81 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v6 1/3] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/t/002_cic_2pc.pl | 238 +++++++++++++++++++++++++++++++
2 files changed, 239 insertions(+), 1 deletion(-)
create mode 100644 contrib/amcheck/t/002_cic_2pc.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..5b5427d193 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -12,7 +12,7 @@ PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
-TAP_TESTS = 1
+TAP_TESTS = 2
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/amcheck/t/002_cic_2pc.pl b/contrib/amcheck/t/002_cic_2pc.pl
new file mode 100644
index 0000000000..541ac2e8a9
--- /dev/null
+++ b/contrib/amcheck/t/002_cic_2pc.pl
@@ -0,0 +1,238 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for point-in-time-recovery (PITR) with prepared transactions
+use strict;
+use warnings;
+
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = get_new_node('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(5);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(5);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+$main_h->pump_nb;
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = background_pgbench('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --time=1');
+
+# Run pgbench.
+pgbench(
+ '--no-vacuum --client=5 --time=1',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ \set txid random(1, 1000000000)
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'tx:txid';
+ COMMIT PREPARED 'tx:txid';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = $pgbench_h->result(0); # Need extra check for ($Config{osname} eq "MSWin32") ?
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
+
+# invoke pgbench, with parameters:
+# $opts: options as a string to be split on spaces
+# $stat: expected exit status
+# $out: reference to a regexp list that must match stdout
+# $err: reference to a regexp list that must match stderr
+# $name: name of test for error messages
+# $files: reference to filename/contents dictionary
+# @args: further raw options or arguments
+sub pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my ($opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $node->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+
+# invoke pgbench in background alike background_psql
+sub background_pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my ($dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $node->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v6-0002-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxid.patch (application/octet-stream, 8.2K) download
From 515f3cd1081880b72a230487a2a9002a5e56fcdc Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v6 2/3] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2pc
---
src/backend/access/transam/twophase.c | 49 +++++++++++++-
src/backend/storage/lmgr/lock.c | 96 +++++++++++++++++++++------
src/include/access/twophase.h | 12 ++++
3 files changed, 134 insertions(+), 23 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..45e1c3259b 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -373,8 +373,10 @@ MarkAsPreparing(TransactionId xid, const char *gid,
TimestampTz prepared_at, Oid owner, Oid databaseid)
{
GlobalTransaction gxact;
+
int i;
+
if (strlen(gid) >= GIDSIZE)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -459,14 +461,16 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +850,45 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+XidListEntry
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc;
+ proc = &ProcGlobal->allProcs[gxact->pgprocno];
+
+ if (proc->backendId == vxid.backendId &&
+ proc->lxid == vxid.localTransactionId)
+ {
+ if (result.xid != InvalidTransactionId)
+ {
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..55efb33a7b 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -3017,7 +3018,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3078,7 +3086,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -4447,6 +4462,51 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * PreparedXactLock
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * fake procarray entries.
+ */
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ XidListEntry xidlist;
+
+ /* Questionable heuristics */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ if (TransactionIdIsValid(xid))
+ {
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /* We must search for vxids in 2pc state */
+ /* XXX: O(N*N) complexity where N is number of prepared xacts */
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ elog(NOTICE,"XXX: Sucessfully found xid by vxid"); // Remove this line
+ }
+
+ while (true)
+ {
+ if (TransactionIdIsValid(xidlist.xid))
+ {
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar != LOCKACQUIRE_NOT_AVAIL)
+ LockRelease(&tag, ShareLock, false);
+ return lar != LOCKACQUIRE_NOT_AVAIL;
+ }
+ if (xidlist.next == NULL)
+ return true;
+ xidlist = *xidlist.next;
+ }
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4519,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return PreparedXactLock(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4544,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4558,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4615,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return PreparedXactLock(vxid, xid, wait);;
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..70b3cb9391 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -58,4 +58,16 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+/*
+ * This struc is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v6-0003-Various-debug-stuff-and-fix-in-xidlist.xid.patch (application/octet-stream, 9.1K) download
From b7fda517fe7041c28b0bbd85ae19bb1a557eafb5 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Thu, 29 Jul 2021 17:26:44 +0500
Subject: [PATCH v6 3/3] Various debug stuff and fix in xidlist.xid
---
contrib/amcheck/verify_nbtree.c | 19 +++++++++++++++++--
src/backend/commands/indexcmds.c | 10 +++++++++-
src/backend/storage/lmgr/lock.c | 30 +++++++++++++++++++-----------
3 files changed, 45 insertions(+), 14 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index fdfc320e84..3e1ac42051 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -2453,6 +2453,8 @@ bt_downlink_missing_check(BtreeCheckState *state, bool rightsplit,
* REINDEX to repair corruption when there was an (undetected) broken HOT chain
* also allows us to detect the corruption in many cases.
*/
+
+#include "storage/bufpage.h"
static void
bt_tuple_present_callback(Relation index, ItemPointer tid, Datum *values,
bool *isnull, bool tupleIsAlive, void *checkstate)
@@ -2471,16 +2473,29 @@ bt_tuple_present_callback(Relation index, ItemPointer tid, Datum *values,
/* Probe Bloom filter -- tuple should be present */
if (bloom_lacks_element(state->filter, (unsigned char *) norm,
IndexTupleSize(norm)))
+ {
+ BlockIdData bid = tid->ip_blkid;
+ Buffer buf = ReadBufferExtended(state->heaprel, MAIN_FORKNUM, BlockIdGetBlockNumber(&bid), RBM_NORMAL, NULL);
+ LockBuffer(buf, BUFFER_LOCK_SHARE);
+ Page page = BufferGetPage(buf);
+
+ HeapTupleHeader tuphdr = (HeapTupleHeader) PageGetItem(page, PageGetItemId(page,tid->ip_posid));
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("heap tuple (%u,%u) from table \"%s\" lacks matching index tuple within index \"%s\"",
+ errmsg("heap tuple (%u,%u) from table \"%s\" lacks matching index tuple within index \"%s\" xmin %d xmax %d",
ItemPointerGetBlockNumber(&(itup->t_tid)),
ItemPointerGetOffsetNumber(&(itup->t_tid)),
RelationGetRelationName(state->heaprel),
- RelationGetRelationName(state->rel)),
+ RelationGetRelationName(state->rel),
+ HeapTupleHeaderGetRawXmin(tuphdr),
+ HeapTupleHeaderGetRawXmax(tuphdr)),
!state->readonly
? errhint("Retrying verification using the function bt_index_parent_check() might provide a more specific error.")
: 0));
+ }
state->heaptuplespresent++;
pfree(itup);
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 76774dce06..51cc484e42 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1536,7 +1536,6 @@ DefineIndex(Oid relationId,
*/
pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
PROGRESS_CREATEIDX_PHASE_WAIT_2);
- WaitForLockers(heaplocktag, ShareLock, true);
/*
* Now take the "reference snapshot" that will be used by validate_index()
@@ -1556,6 +1555,8 @@ DefineIndex(Oid relationId,
snapshot = RegisterSnapshot(GetTransactionSnapshot());
PushActiveSnapshot(snapshot);
+ WaitForLockers(heaplocktag, ShareLock, true);
+
/*
* Scan the index and the heap, insert any missing index entries.
*/
@@ -3519,6 +3520,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* for efficiency.
*/
+ elog(WARNING, "Phase 1");
/*
* Phase 1 of REINDEX CONCURRENTLY
*
@@ -3669,6 +3671,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* to set the PROC_IN_SAFE_IC flag here.
*/
+ elog(WARNING, "Phase 2");
/*
* Phase 2 of REINDEX CONCURRENTLY
*
@@ -3730,6 +3733,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* need to set the PROC_IN_SAFE_IC flag here.
*/
+ elog(WARNING, "Phase 3");
/*
* Phase 3 of REINDEX CONCURRENTLY
*
@@ -3816,6 +3820,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
CommitTransactionCommand();
}
+ elog(WARNING, "Phase 4");
/*
* Phase 4 of REINDEX CONCURRENTLY
*
@@ -3889,6 +3894,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* done, and that lasts for a very short period.
*/
+ elog(WARNING, "Phase 5");
/*
* Phase 5 of REINDEX CONCURRENTLY
*
@@ -3925,6 +3931,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* done, and that lasts for a very short period.
*/
+ elog(WARNING, "Phase 6");
/*
* Phase 6 of REINDEX CONCURRENTLY
*
@@ -3962,6 +3969,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
PopActiveSnapshot();
CommitTransactionCommand();
+ elog(WARNING, "Phase Final");
/*
* Finally, release the session-level lock on the table.
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 55efb33a7b..3644d3f4cf 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -3019,13 +3019,13 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
/* Prefer real Xid over local Xid */
- if (TransactionIdIsValid(proc->xid))
+ /*if (TransactionIdIsValid(proc->xid))
{
vxids[count].backendId = InvalidBackendId;
vxids[count].localTransactionId = proc->xid;
count++;
}
- else if (VirtualTransactionIdIsValid(vxid))
+ else */if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3087,13 +3087,13 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
/* Prefer real Xid over local Xid */
- if (TransactionIdIsValid(proc->xid))
+ /*if (TransactionIdIsValid(proc->xid))
{
vxids[count].backendId = InvalidBackendId;
vxids[count].localTransactionId = proc->xid;
count++;
}
- else if (VirtualTransactionIdIsValid(vxid))
+ else */if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -4468,7 +4468,7 @@ VirtualXactLockTableCleanup(void)
* Wait for xid completition if have xid. Otherwise try to find xid among
* fake procarray entries.
*/
-static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait)
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xidx, bool wait)
{
LockAcquireResult lar;
LOCKTAG tag;
@@ -4477,10 +4477,11 @@ static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool
/* Questionable heuristics */
if (max_prepared_xacts == 0)
return true;
+ elog(WARNING,"XXX: PreparedXactLock xid %d, vxid %d/%d",xidx, vxid.backendId, vxid.localTransactionId); // Remove this line
- if (TransactionIdIsValid(xid))
+ if (TransactionIdIsValid(xidx))
{
- xidlist.xid = xid;
+ xidlist.xid = xidx;
xidlist.next = NULL;
}
else
@@ -4488,18 +4489,19 @@ static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool
/* We must search for vxids in 2pc state */
/* XXX: O(N*N) complexity where N is number of prepared xacts */
xidlist = TwoPhaseGetXidByVXid(vxid);
- elog(NOTICE,"XXX: Sucessfully found xid by vxid"); // Remove this line
+ elog(WARNING,"XXX: Sucessfully found xid by vxid %d", xidlist.xid); // Remove this line
}
while (true)
{
if (TransactionIdIsValid(xidlist.xid))
{
- SET_LOCKTAG_TRANSACTION(tag, xid);
+ SET_LOCKTAG_TRANSACTION(tag, xidlist.xid);
lar = LockAcquire(&tag, ShareLock, false, !wait);
if (lar != LOCKACQUIRE_NOT_AVAIL)
LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ return false;
}
if (xidlist.next == NULL)
return true;
@@ -4524,6 +4526,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ elog(WARNING,"XXX: VirtualXactLock vxid %d/%d", vxid.backendId, vxid.localTransactionId); // Remove this line
/*
* Already prepared transactions don't hold vxid locks. The
@@ -4544,7 +4547,10 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
+ {
+ elog(WARNING,"Backend is gone");
return PreparedXactLock(vxid, InvalidTransactionId, wait);
+ }
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4557,12 +4563,13 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
if (proc->backendId != vxid.backendId
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
+ elog(WARNING,"Backend is doing something else");
LWLockRelease(&proc->fpInfoLock);
return PreparedXactLock(vxid, InvalidTransactionId, wait);
}
/* Save the xid to test if transaction coverted to 2pc later */
- xid = proc->xid;
+ xid = proc->xid;
/*
* If we aren't asked to wait, there's no need to set up a lock table
@@ -4615,6 +4622,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
+ elog(WARNING,"Finally waiting on xid %d", xid);
return PreparedXactLock(vxid, xid, wait);;
}
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-29 14:21 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-29 14:21 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 29 июля 2021 г., в 17:30, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> Searching further.
Currently CIC test breaks indexes even without 2PC. How is it supposed to work if vxid stays vxid through GetLockConflicts()\WaitForLockersMultiple() barrier and then suddenly converts to xid and commits before index validated?
Best regards, Andrey Borodin.
Attachments:
v7-0001-Do-not-use-Prepared-transactions-in-CIC-test.patch (application/octet-stream, 6.5K) download
From 369818bc9e43df3e457950b04d424343a54f3ce7 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Thu, 29 Jul 2021 19:19:09 +0500
Subject: [PATCH v7] Do not use Prepared transactions in CIC test
---
contrib/amcheck/t/003_break_cic.pl | 242 +++++++++++++++++++++++++++++
1 file changed, 242 insertions(+)
create mode 100644 contrib/amcheck/t/003_break_cic.pl
diff --git a/contrib/amcheck/t/003_break_cic.pl b/contrib/amcheck/t/003_break_cic.pl
new file mode 100644
index 0000000000..6ab1ff76f7
--- /dev/null
+++ b/contrib/amcheck/t/003_break_cic.pl
@@ -0,0 +1,242 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for point-in-time-recovery (PITR) with prepared transactions
+use strict;
+use warnings;
+
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = get_new_node('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(5);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(5);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+--disable test
+--REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+$main_h->pump_nb;
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+#$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = background_pgbench('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --time=1');
+
+# Run pgbench.
+pgbench(
+ '--no-vacuum --client=5 --time=1',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ \set txid random(1, 1000000000)
+ BEGIN;
+ select pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ --select pg_sleep(0.001);
+ COMMIT;
+ --PREPARE TRANSACTION 'tx:txid';
+ --COMMIT PREPARED 'tx:txid';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = $pgbench_h->result(0); # Need extra check for ($Config{osname} eq "MSWin32") ?
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
+
+# invoke pgbench, with parameters:
+# $opts: options as a string to be split on spaces
+# $stat: expected exit status
+# $out: reference to a regexp list that must match stdout
+# $err: reference to a regexp list that must match stderr
+# $name: name of test for error messages
+# $files: reference to filename/contents dictionary
+# @args: further raw options or arguments
+sub pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my ($opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $node->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+
+# invoke pgbench in background alike background_psql
+sub background_pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my ($dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $node->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-30 02:25 ` Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-07-30 02:25 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Thu, Jul 29, 2021 at 07:21:44PM +0500, Andrey Borodin wrote:
> Currently CIC test breaks indexes even without 2PC. How is it supposed to work if vxid stays vxid through GetLockConflicts()\WaitForLockersMultiple() barrier and then suddenly converts to xid and commits before index validated?
It's an inval race condition. Here's the sequence of events leading to a
failure. "i:" lines are events within the INSERT backend, and "r:" lines are
events within the REINDEX CONCURRENTLY backend:
r: Phase 2 begins.
i: Start INSERT.
i: Acquire RowExclusiveLock on "idx_ccnew".
i: RelationBuildDesc() begins for relation "idx_ccnew".
i: RelationBuildDesc() calls RelationInitIndexAccessInfo(), which loads
indisready=f.
r: Phase 2 commits indisready=t for idx_ccnew.
r: Start waiting for the INSERT to finish.
i: LocalExecuteInvalidationMessage() calls RelationCacheInvalidateEntry() for
the OID of relation "idx_ccnew". Its RelationIdCacheLookup() call returns
NULL, because nothing has passed the relation to RelationCacheInsert().
i: RelationBuildDesc(), the same call that began above, calls
RelationCacheInsert(). This is a problem, because the entry needs
invalidation, but we already consumed the invalidation messages.
i: Inserts heap tuple with xmin=xid1. Does not insert to idx_ccnew. Commits.
r: validate_index() runs, adding the index tuple for xid1.
i: Second INSERT runs, adding a heap tuple with xmin=xid2. It uses the stale
relcache entry having indisready=f, so this insert also does not modify
idx_ccnew. This time, it was responsible for making that index entry;
nothing else will.
In summary, the relcache is deaf to invals of a given relation during
RelationBuildDesc() of that relation. One fix would be to make
RelationBuildDesc() more like this pseudocode:
retry:
WatchInvalFor(targetRelId);
... complicated build steps that may call AcceptInvalidationMessages() ...
if (ReceivedInvalFor(targetRelId))
goto retry;
I'm attaching a proof-of-concept sufficient to let your v7 pgbench test pass.
Among other things, I haven't checked whether other inval message types have
the same hazard. What alternative fix designs should we consider?
Attachments:
inval-build-race-v0.patch (text/x-diff, 1.7K) download
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 13d9994af3..e5f8809445 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1228,10 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ inProgress[in_progress_offset].relid = InvalidOid;
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2805,6 +2824,13 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-30 10:42 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-30 10:42 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 30 июля 2021 г., в 07:25, Noah Misch <noah@leadboat.com> написал(а):
> What alternative fix designs should we consider?
I observe that provided patch fixes CIC under normal transactions, but test with 2PC still fails similarly.
Unindexed tuple was committed somewhere at the end of Phase 3 or 4.
2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl LOG: statement: REINDEX INDEX CONCURRENTLY idx;
2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: Phase 1
2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: Phase 2
2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6735
2021-07-30 15:35:31.807 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6736
2021-07-30 15:35:31.808 +05 [25987] 002_cic_2pc.pl WARNING: Phase 3
2021-07-30 15:35:31.808 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6750
2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: Phase 4
2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: Phase 5
2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6762
2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6763
2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: Phase 6
2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid 6/2166
2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6767
2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6764
2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6765
2021-07-30 15:35:31.811 +05 [25987] 002_cic_2pc.pl WARNING: Phase Final
2021-07-30 15:35:31.811 +05 [25987] 002_cic_2pc.pl LOG: statement: SELECT bt_index_check('idx',true);
2021-07-30 15:35:31.813 +05 [25987] 002_cic_2pc.pl ERROR: heap tuple (46,16) from table "tbl" lacks matching index tuple within index "idx" xmin 6751 xmax 0
Attaching debug logging patch, amcheck is modified to return xmin. Trace is gathered by "grep -e ERROR -e REINDEX -e WARN -e SELECT tmp_check/log/002_cic_2pc_CIC_2PC_test.log".
How deep the rabbit hole goes?
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (4 files, .tar.gz)
v8-0001-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 7.0K) download
From 6edccb3906472e88b57f7642d496c42078fddc81 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v8 1/4] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/t/002_cic_2pc.pl | 238 +++++++++++++++++++++++++++++++
2 files changed, 239 insertions(+), 1 deletion(-)
create mode 100644 contrib/amcheck/t/002_cic_2pc.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..5b5427d193 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -12,7 +12,7 @@ PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
-TAP_TESTS = 1
+TAP_TESTS = 2
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/amcheck/t/002_cic_2pc.pl b/contrib/amcheck/t/002_cic_2pc.pl
new file mode 100644
index 0000000000..541ac2e8a9
--- /dev/null
+++ b/contrib/amcheck/t/002_cic_2pc.pl
@@ -0,0 +1,238 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for point-in-time-recovery (PITR) with prepared transactions
+use strict;
+use warnings;
+
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = get_new_node('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(5);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(5);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+$main_h->pump_nb;
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = background_pgbench('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --time=1');
+
+# Run pgbench.
+pgbench(
+ '--no-vacuum --client=5 --time=1',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ \set txid random(1, 1000000000)
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'tx:txid';
+ COMMIT PREPARED 'tx:txid';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = $pgbench_h->result(0); # Need extra check for ($Config{osname} eq "MSWin32") ?
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
+
+# invoke pgbench, with parameters:
+# $opts: options as a string to be split on spaces
+# $stat: expected exit status
+# $out: reference to a regexp list that must match stdout
+# $err: reference to a regexp list that must match stderr
+# $name: name of test for error messages
+# $files: reference to filename/contents dictionary
+# @args: further raw options or arguments
+sub pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my ($opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $node->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+
+# invoke pgbench in background alike background_psql
+sub background_pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my ($dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $node->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v8-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcach.patch (application/octet-stream, 2.1K) download
From 639d77fd7a32f094156d1171dc496ee33a296a5c Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 14:40:16 +0500
Subject: [PATCH v8 2/4] PoC fix for race in RelationBuildDesc() and relcache
invalidation
---
src/backend/utils/cache/relcache.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fd05615e76..c7e9393612 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1228,10 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ inProgress[in_progress_offset].relid = InvalidOid;
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2802,6 +2821,13 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
--
2.24.3 (Apple Git-128)
=
v8-0003-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxid.patch (application/octet-stream, 8.5K) download
From 9000a4cb36ee6480b2c841edd7f14058d1716099 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v8 3/4] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2pc
---
src/backend/access/transam/twophase.c | 46 +++++++++-
src/backend/storage/lmgr/lock.c | 119 +++++++++++++++++++++-----
src/include/access/twophase.h | 13 +++
3 files changed, 155 insertions(+), 23 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..55c68364ce 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +847,45 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+XidListEntry
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc;
+ proc = &ProcGlobal->allProcs[gxact->pgprocno];
+
+ if (proc->backendId == vxid.backendId &&
+ proc->lxid == vxid.localTransactionId)
+ {
+ if (result.xid != InvalidTransactionId)
+ {
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..becddef7fe 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -3017,7 +3018,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3078,7 +3086,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -4447,6 +4462,74 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * PreparedXactLock
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * fake procarray entries.
+ */
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ XidListEntry xidlist;
+ XidListEntry *xidlist_ptr = NULL; /* pointer to TwoPhaseGetXidByVXid()s pallocs */
+ bool result;
+
+ /* Questionable heuristics */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ if (TransactionIdIsValid(xid))
+ {
+ /* We know exact xid - no need to search in 2PC state */
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /* We must search for vxids in 2pc state */
+ /* XXX: O(N*N) complexity where N is number of prepared xacts */
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ /* Return if transaction is gone entirely */
+ if (!TransactionIdIsValid(xidlist.xid))
+ return true;
+ xidlist_ptr = xidlist.next;
+ }
+
+
+ while (true)
+ {
+ /* Iterate over possible xids */
+ Assert(TransactionIdIsValid(xidlist.xid));
+ SET_LOCKTAG_TRANSACTION(tag, xidlist.xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ {
+ result = false;
+ break;
+ }
+
+ LockRelease(&tag, ShareLock, false);
+
+ if (xidlist.next == NULL)
+ {
+ result = true;
+ break;
+ }
+ xidlist = *xidlist.next;
+ }
+
+ /* Free all pallocs made by TwoPhaseGetXidByVXid() */
+ while (xidlist_ptr)
+ {
+ void *chunk = xidlist_ptr;
+ xidlist_ptr = xidlist_ptr->next;
+ pfree(chunk);
+ }
+ return result;
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4542,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return PreparedXactLock(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4567,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4581,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4638,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return PreparedXactLock(vxid, xid, wait);;
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..ba0ac17359 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -25,6 +25,17 @@
*/
typedef struct GlobalTransactionData *GlobalTransaction;
+/*
+ * XidListEntry is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+
/* GUC variable */
extern PGDLLIMPORT int max_prepared_xacts;
@@ -58,4 +69,6 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v8-0004-Debug-suff.patch (application/octet-stream, 4.3K) download
From 8da372b5bbae38d162a01edc63094f9099e138e8 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 15:06:30 +0500
Subject: [PATCH v8 4/4] Debug suff
---
contrib/amcheck/verify_nbtree.c | 17 +++++++++++++++--
src/backend/commands/indexcmds.c | 8 +++++++-
src/backend/storage/lmgr/lock.c | 1 +
3 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index fdfc320e84..ac2cfe19c6 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -2471,16 +2471,29 @@ bt_tuple_present_callback(Relation index, ItemPointer tid, Datum *values,
/* Probe Bloom filter -- tuple should be present */
if (bloom_lacks_element(state->filter, (unsigned char *) norm,
IndexTupleSize(norm)))
+{
+ BlockIdData bid = tid->ip_blkid;
+ Buffer buf = ReadBufferExtended(state->heaprel, MAIN_FORKNUM, BlockIdGetBlockNumber(&bid), RBM_NORMAL, NULL);
+ LockBuffer(buf, BUFFER_LOCK_SHARE);
+ Page page = BufferGetPage(buf);
+
+ HeapTupleHeader tuphdr = (HeapTupleHeader) PageGetItem(page, PageGetItemId(page,tid->ip_posid));
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("heap tuple (%u,%u) from table \"%s\" lacks matching index tuple within index \"%s\"",
+ errmsg("heap tuple (%u,%u) from table \"%s\" lacks matching index tuple within index \"%s\" xmin %d xmax %d",
ItemPointerGetBlockNumber(&(itup->t_tid)),
ItemPointerGetOffsetNumber(&(itup->t_tid)),
RelationGetRelationName(state->heaprel),
- RelationGetRelationName(state->rel)),
+ RelationGetRelationName(state->rel),
+ HeapTupleHeaderGetRawXmin(tuphdr),
+ HeapTupleHeaderGetRawXmax(tuphdr)),
!state->readonly
? errhint("Retrying verification using the function bt_index_parent_check() might provide a more specific error.")
: 0));
+ }
state->heaptuplespresent++;
pfree(itup);
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 76774dce06..8a4a17d81b 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -3519,6 +3519,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* for efficiency.
*/
+ elog(WARNING, "Phase 1");
/*
* Phase 1 of REINDEX CONCURRENTLY
*
@@ -3668,7 +3669,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* Because we don't take a snapshot in this transaction, there's no need
* to set the PROC_IN_SAFE_IC flag here.
*/
-
+ elog(WARNING, "Phase 2");
/*
* Phase 2 of REINDEX CONCURRENTLY
*
@@ -3730,6 +3731,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* need to set the PROC_IN_SAFE_IC flag here.
*/
+ elog(WARNING, "Phase 3");
/*
* Phase 3 of REINDEX CONCURRENTLY
*
@@ -3816,6 +3818,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
CommitTransactionCommand();
}
+ elog(WARNING, "Phase 4");
/*
* Phase 4 of REINDEX CONCURRENTLY
*
@@ -3889,6 +3892,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* done, and that lasts for a very short period.
*/
+ elog(WARNING, "Phase 5");
/*
* Phase 5 of REINDEX CONCURRENTLY
*
@@ -3925,6 +3929,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
* done, and that lasts for a very short period.
*/
+ elog(WARNING, "Phase 6");
/*
* Phase 6 of REINDEX CONCURRENTLY
*
@@ -4010,6 +4015,7 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextDelete(private_context);
pgstat_progress_end_command();
+ elog(WARNING, "Phase Final");
return true;
}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index becddef7fe..e56bee66bd 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -4547,6 +4547,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ elog(WARNING,"XXX: VirtualXactLock vxid %d/%d", vxid.backendId, vxid.localTransactionId); // Remove this line
/*
* Already prepared transactions don't hold vxid locks. The
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-30 18:41 ` Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-07-30 18:41 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Fri, Jul 30, 2021 at 03:42:10PM +0500, Andrey Borodin wrote:
> > 30 июля 2021 г., в 07:25, Noah Misch <noah@leadboat.com> написал(а):
> > What alternative fix designs should we consider?
>
> I observe that provided patch fixes CIC under normal transactions, but test with 2PC still fails similarly.
> Unindexed tuple was committed somewhere at the end of Phase 3 or 4.
> 2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl LOG: statement: REINDEX INDEX CONCURRENTLY idx;
> 2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: Phase 1
> 2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: Phase 2
> 2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6735
> 2021-07-30 15:35:31.807 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6736
> 2021-07-30 15:35:31.808 +05 [25987] 002_cic_2pc.pl WARNING: Phase 3
> 2021-07-30 15:35:31.808 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6750
> 2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: Phase 4
> 2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: Phase 5
> 2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6762
> 2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6763
> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: Phase 6
> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid 6/2166
> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6767
> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6764
> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6765
> 2021-07-30 15:35:31.811 +05 [25987] 002_cic_2pc.pl WARNING: Phase Final
> 2021-07-30 15:35:31.811 +05 [25987] 002_cic_2pc.pl LOG: statement: SELECT bt_index_check('idx',true);
> 2021-07-30 15:35:31.813 +05 [25987] 002_cic_2pc.pl ERROR: heap tuple (46,16) from table "tbl" lacks matching index tuple within index "idx" xmin 6751 xmax 0
I see a failure, too. Once again, "i:" lines are events within the INSERT
backend, and "r:" lines are events within the REINDEX CONCURRENTLY backend:
r: Phase 2 begins.
i: INSERT. Start PREPARE.
r: Phase 2 commits indisready=t for idx_ccnew.
r: Start waiting for the INSERT to finish.
i: PREPARE finishes.
r: Wake up and start validate_index(). This is a problem. It needed to wait
for COMMIT PREPARED to finish.
This may have a different explanation than the failure you saw, because my
INSERT transaction already had a permanent XID before the start of phase 3. I
won't have time to study this further in the next several days. Can you find
out where things go wrong? The next thing I would study is VirtualXactLock(),
specifically what happens if the lock holder is a normal backend (with or
without an XID) when VirtualXactLock() starts but becomes a prepared
transaction (w/ different PGPROC) before VirtualXactLock() ends.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-07-31 08:37 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-07-31 08:37 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 30 июля 2021 г., в 23:41, Noah Misch <noah@leadboat.com> написал(а):
>
> On Fri, Jul 30, 2021 at 03:42:10PM +0500, Andrey Borodin wrote:
>>> 30 июля 2021 г., в 07:25, Noah Misch <noah@leadboat.com> написал(а):
>>> What alternative fix designs should we consider?
>>
>> I observe that provided patch fixes CIC under normal transactions, but test with 2PC still fails similarly.
>> Unindexed tuple was committed somewhere at the end of Phase 3 or 4.
>> 2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl LOG: statement: REINDEX INDEX CONCURRENTLY idx;
>> 2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: Phase 1
>> 2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: Phase 2
>> 2021-07-30 15:35:31.806 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6735
>> 2021-07-30 15:35:31.807 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6736
>> 2021-07-30 15:35:31.808 +05 [25987] 002_cic_2pc.pl WARNING: Phase 3
>> 2021-07-30 15:35:31.808 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6750
>> 2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: Phase 4
>> 2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: Phase 5
>> 2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6762
>> 2021-07-30 15:35:31.809 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6763
>> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: Phase 6
>> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid 6/2166
>> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6767
>> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6764
>> 2021-07-30 15:35:31.810 +05 [25987] 002_cic_2pc.pl WARNING: XXX: VirtualXactLock vxid -1/6765
>> 2021-07-30 15:35:31.811 +05 [25987] 002_cic_2pc.pl WARNING: Phase Final
>> 2021-07-30 15:35:31.811 +05 [25987] 002_cic_2pc.pl LOG: statement: SELECT bt_index_check('idx',true);
>> 2021-07-30 15:35:31.813 +05 [25987] 002_cic_2pc.pl ERROR: heap tuple (46,16) from table "tbl" lacks matching index tuple within index "idx" xmin 6751 xmax 0
>
> I see a failure, too. Once again, "i:" lines are events within the INSERT
> backend, and "r:" lines are events within the REINDEX CONCURRENTLY backend:
>
> r: Phase 2 begins.
> i: INSERT. Start PREPARE.
> r: Phase 2 commits indisready=t for idx_ccnew.
> r: Start waiting for the INSERT to finish.
> i: PREPARE finishes.
> r: Wake up and start validate_index(). This is a problem. It needed to wait
> for COMMIT PREPARED to finish.
I'l investigate this scenario. I've tried to sprinkle some more WaitForLockersMultiple() yet without success.
> This may have a different explanation than the failure you saw, because my
> INSERT transaction already had a permanent XID before the start of phase 3. I
> won't have time to study this further in the next several days. Can you find
> out where things go wrong?
I'll try. This bug is #1 priority for me. We repack ~pb of indexes each weekend (only bloated, many in fact are bloated). And seems like they all are endangered.
> The next thing I would study is VirtualXactLock(),
> specifically what happens if the lock holder is a normal backend (with or
> without an XID) when VirtualXactLock() starts but becomes a prepared
> transaction (w/ different PGPROC) before VirtualXactLock() ends.
PreparedXactLock() will do the trick. If we have xid - we always take a lock on xid. If we have vxid - we try to convert it to xid and look in all PGPROCs for 2PCs. And then again - wait for xid.
At this point I'm certain that if any transaction is reported by GetLockConflicts() it will get awaited by VirtualXactLock().
The problem is that rogue transaction was never reported by GetLockConflicts().
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-07-31 08:51 ` Peter Geoghegan <pg@bowt.ie>
2021-08-02 12:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 107+ messages in thread
From: Peter Geoghegan @ 2021-07-31 08:51 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Sat, Jul 31, 2021 at 11:37 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> I'll try. This bug is #1 priority for me. We repack ~pb of indexes each weekend (only bloated, many in fact are bloated). And seems like they all are endangered.
I have a tip, which might make life a little easier: maybe use rr for this? See:
https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD#...
Having a stable recording with fixed logical times *vastly* simplifies
debugging in practice. I found this very useful when debugging the SSI
Jepsen bug, which was a complex race condition that could only be
reproduced through a complex procedure. Which is also what we have
here now, more or less. Your personal notes on the bug can sensibly
track specific pointer values, logical times, etc.
You can even "pack" a recording, making it into a self-contained thing
that doesn't rely on having the original binaries. This makes the
recording stable enough to keep for weeks or months, share with other
people, etc.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
@ 2021-08-02 12:45 ` Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 107+ messages in thread
From: Andrey Borodin @ 2021-08-02 12:45 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 31 июля 2021 г., в 13:51, Peter Geoghegan <pg@bowt.ie> написал(а):
>
> maybe use rr for this?
Thanks, Peter!
I'll give it a try.
Currently I observe that during PrepareTransaction() there might be a moment when the transaction is not reported by GetLockConflicts() with relevant lock.
I observe that PrepareTransaction() for violating xid is somewhere between
CallXactCallbacks(XACT_EVENT_PREPARE);
and
PostPrepare_PgStat();
And GetLockConflicts() fails to detect this transaction.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
@ 2021-08-06 18:03 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-06 18:03 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 31 июля 2021 г., в 13:51, Peter Geoghegan <pg@bowt.ie> написал(а):
>
> maybe use rr for this?
rr helps to explain why something happened.
But how to get why something did not happen?
During GetLockConflicts() proclocks SHMQueue do not include locks of concurrently running xid within PrepareTransaction() [seems like it's with PostPrepare_locks(), but I'm not sure].
The problem is reproducible within 1 second by the script leading to amcheck-detected corruption.
I'd appreciate one more hint...
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-07 15:33 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-07 15:33 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 6 авг. 2021 г., в 23:03, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> But how to get why something did not happen?
> During GetLockConflicts() proclocks SHMQueue do not include locks of concurrently running xid within PrepareTransaction() [seems like it's with PostPrepare_locks(), but I'm not sure].
> The problem is reproducible within 1 second by the script leading to amcheck-detected corruption.
Ok, finally I've figured out this charade.
AtPrepare_Locks() transfers fast-path locks to normal locks pointing to PGPROC with xid in a process of conversion to 2PC.
ProcArrayClearTransaction(MyProc) resets xid int PGROCs.
GetLockConflicts() sees empty xid and vxid.
PostPrepare_Locks(xid) hids things with new PGPROC for 2PC.
PFA PoC fix. But I'm entirely not sure it's OK to do reorder things at PrepareTransaction() this way.
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (4 files, .tar.gz)
v9-0001-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 7.0K) download
From 6edccb3906472e88b57f7642d496c42078fddc81 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v9 1/4] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/t/002_cic_2pc.pl | 238 +++++++++++++++++++++++++++++++
2 files changed, 239 insertions(+), 1 deletion(-)
create mode 100644 contrib/amcheck/t/002_cic_2pc.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..5b5427d193 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -12,7 +12,7 @@ PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
-TAP_TESTS = 1
+TAP_TESTS = 2
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/amcheck/t/002_cic_2pc.pl b/contrib/amcheck/t/002_cic_2pc.pl
new file mode 100644
index 0000000000..541ac2e8a9
--- /dev/null
+++ b/contrib/amcheck/t/002_cic_2pc.pl
@@ -0,0 +1,238 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for point-in-time-recovery (PITR) with prepared transactions
+use strict;
+use warnings;
+
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = get_new_node('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(5);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(5);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+$main_h->pump_nb;
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = background_pgbench('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --time=1');
+
+# Run pgbench.
+pgbench(
+ '--no-vacuum --client=5 --time=1',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ \set txid random(1, 1000000000)
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'tx:txid';
+ COMMIT PREPARED 'tx:txid';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = $pgbench_h->result(0); # Need extra check for ($Config{osname} eq "MSWin32") ?
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
+
+# invoke pgbench, with parameters:
+# $opts: options as a string to be split on spaces
+# $stat: expected exit status
+# $out: reference to a regexp list that must match stdout
+# $err: reference to a regexp list that must match stderr
+# $name: name of test for error messages
+# $files: reference to filename/contents dictionary
+# @args: further raw options or arguments
+sub pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my ($opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $node->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+
+# invoke pgbench in background alike background_psql
+sub background_pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my ($dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $node->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v9-0004-PoC-fix-clear-xid.patch (application/octet-stream, 1.0K) download
From fb693a0b65a4967232e0ad75aef183c2957cbd0a Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 7 Aug 2021 20:27:14 +0500
Subject: [PATCH v9 4/4] PoC fix clear xid
---
src/backend/access/transam/xact.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 441445927e..b24a3de8d4 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2500,7 +2500,6 @@ PrepareTransaction(void)
* done *after* the prepared transaction has been marked valid, else
* someone may think it is unlocked and recyclable.
*/
- ProcArrayClearTransaction(MyProc);
/*
* In normal commit-processing, this is all non-critical post-transaction
@@ -2535,6 +2534,8 @@ PrepareTransaction(void)
PostPrepare_MultiXact(xid);
PostPrepare_Locks(xid);
+
+ ProcArrayClearTransaction(MyProc);
PostPrepare_PredicateLocks(xid);
ResourceOwnerRelease(TopTransactionResourceOwner,
--
2.24.3 (Apple Git-128)
=
v9-0003-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxid.patch (application/octet-stream, 8.5K) download
From 9000a4cb36ee6480b2c841edd7f14058d1716099 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v9 3/4] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2pc
---
src/backend/access/transam/twophase.c | 46 +++++++++-
src/backend/storage/lmgr/lock.c | 119 +++++++++++++++++++++-----
src/include/access/twophase.h | 13 +++
3 files changed, 155 insertions(+), 23 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..55c68364ce 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +847,45 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+XidListEntry
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc;
+ proc = &ProcGlobal->allProcs[gxact->pgprocno];
+
+ if (proc->backendId == vxid.backendId &&
+ proc->lxid == vxid.localTransactionId)
+ {
+ if (result.xid != InvalidTransactionId)
+ {
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..becddef7fe 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -3017,7 +3018,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3078,7 +3086,14 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
- if (VirtualTransactionIdIsValid(vxid))
+ /* Prefer real Xid over local Xid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxids[count].backendId = InvalidBackendId;
+ vxids[count].localTransactionId = proc->xid;
+ count++;
+ }
+ else if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -4447,6 +4462,74 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * PreparedXactLock
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * fake procarray entries.
+ */
+static bool PreparedXactLock(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ XidListEntry xidlist;
+ XidListEntry *xidlist_ptr = NULL; /* pointer to TwoPhaseGetXidByVXid()s pallocs */
+ bool result;
+
+ /* Questionable heuristics */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ if (TransactionIdIsValid(xid))
+ {
+ /* We know exact xid - no need to search in 2PC state */
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /* We must search for vxids in 2pc state */
+ /* XXX: O(N*N) complexity where N is number of prepared xacts */
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ /* Return if transaction is gone entirely */
+ if (!TransactionIdIsValid(xidlist.xid))
+ return true;
+ xidlist_ptr = xidlist.next;
+ }
+
+
+ while (true)
+ {
+ /* Iterate over possible xids */
+ Assert(TransactionIdIsValid(xidlist.xid));
+ SET_LOCKTAG_TRANSACTION(tag, xidlist.xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ {
+ result = false;
+ break;
+ }
+
+ LockRelease(&tag, ShareLock, false);
+
+ if (xidlist.next == NULL)
+ {
+ result = true;
+ break;
+ }
+ xidlist = *xidlist.next;
+ }
+
+ /* Free all pallocs made by TwoPhaseGetXidByVXid() */
+ while (xidlist_ptr)
+ {
+ void *chunk = xidlist_ptr;
+ xidlist_ptr = xidlist_ptr->next;
+ pfree(chunk);
+ }
+ return result;
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4542,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return PreparedXactLock(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4567,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4581,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return PreparedXactLock(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4638,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return PreparedXactLock(vxid, xid, wait);;
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..ba0ac17359 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -25,6 +25,17 @@
*/
typedef struct GlobalTransactionData *GlobalTransaction;
+/*
+ * XidListEntry is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+
/* GUC variable */
extern PGDLLIMPORT int max_prepared_xacts;
@@ -58,4 +69,6 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v9-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcach.patch (application/octet-stream, 2.1K) download
From 639d77fd7a32f094156d1171dc496ee33a296a5c Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 14:40:16 +0500
Subject: [PATCH v9 2/4] PoC fix for race in RelationBuildDesc() and relcache
invalidation
---
src/backend/utils/cache/relcache.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fd05615e76..c7e9393612 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1228,10 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ inProgress[in_progress_offset].relid = InvalidOid;
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2802,6 +2821,13 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-07 19:00 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-07 19:00 UTC (permalink / raw)
To: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; +Cc: Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
> 7 авг. 2021 г., в 20:33, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> <v9-0001-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch>
> <v9-0004-PoC-fix-clear-xid.patch>
> <v9-0003-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxid.patch>
> <v9-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcach.patch>
Here's v10.
Changes:
1. Added assert in step 2 (fix for missed invalidation message). I wonder how deep possibly could be RelationBuildDesc() inside RelationBuildDesc() inside RelationBuildDesc() ... ? If the depth is unlimited we, probably, need a better data structure.
2. Editorialised step 3 (vxid->xid lookup). Fixing typos and some small bugs.
Tomorrow I'll try to cleanup step 1 (tap tests).
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (4 files, .tar.gz)
v10-0004-PoC-fix-clear-xid.patch (application/octet-stream, 1.0K) download
From 675e62c23535c6c4daf385383a6bc2f3569cf6eb Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 7 Aug 2021 20:27:14 +0500
Subject: [PATCH v10 4/4] PoC fix clear xid
---
src/backend/access/transam/xact.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 441445927e..b24a3de8d4 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2500,7 +2500,6 @@ PrepareTransaction(void)
* done *after* the prepared transaction has been marked valid, else
* someone may think it is unlocked and recyclable.
*/
- ProcArrayClearTransaction(MyProc);
/*
* In normal commit-processing, this is all non-critical post-transaction
@@ -2535,6 +2534,8 @@ PrepareTransaction(void)
PostPrepare_MultiXact(xid);
PostPrepare_Locks(xid);
+
+ ProcArrayClearTransaction(MyProc);
PostPrepare_PredicateLocks(xid);
ResourceOwnerRelease(TopTransactionResourceOwner,
--
2.24.3 (Apple Git-128)
=
v10-0003-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxi.patch (application/octet-stream, 8.4K) download
From b429a1d1ead85bdd33a334a36a3a4dd034ccfedc Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v10 3/4] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2pc
---
src/backend/access/transam/twophase.c | 47 ++++++++++-
src/backend/storage/lmgr/lock.c | 115 ++++++++++++++++++++++----
src/include/access/twophase.h | 13 +++
3 files changed, 154 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..ef551dd0d3 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +847,46 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+XidListEntry
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
+ VirtualTransactionId proc_vxid;
+ GET_VXID_FROM_PGPROC(vxid, *proc);
+
+ if (VirtualTransactionIdEquals(vxid, proc_vxid))
+ {
+ if (result.xid != InvalidTransactionId)
+ {
+ /* Already has candidate - need to alloc some space */
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..c30ff85283 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -3017,6 +3018,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
+ /* Prefer real Xid over local VXid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxid.backendId = InvalidBackendId;
+ vxid.localTransactionId = proc->xid;
+ }
if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3078,6 +3085,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
+ /* Prefer real Xid over local VXid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxid.backendId = InvalidBackendId;
+ vxid.localTransactionId = proc->xid;
+ }
if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -4447,6 +4460,76 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * WaitXact
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * two-phase procarray entries.
+ */
+static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ XidListEntry xidlist;
+ XidListEntry *xidlist_ptr = NULL; /* pointer to TwoPhaseGetXidByVXid()s pallocs */
+ bool result;
+
+ /* Questionable heuristics */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ if (TransactionIdIsValid(xid))
+ {
+ /* We know exact xid - no need to search in 2PC state */
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /*
+ * We must search for vxids in 2pc state
+ * XXX: O(N*N) complexity where N is number of prepared xacts
+ */
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ /* Return if transaction is gone entirely */
+ if (!TransactionIdIsValid(xidlist.xid))
+ return true;
+ xidlist_ptr = xidlist.next;
+ }
+
+
+ while (true)
+ {
+ /* Iterate over possible xids */
+ Assert(TransactionIdIsValid(xidlist.xid));
+ SET_LOCKTAG_TRANSACTION(tag, xidlist.xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ {
+ result = false;
+ break;
+ }
+
+ LockRelease(&tag, ShareLock, false);
+
+ if (xidlist.next == NULL)
+ {
+ result = true;
+ break;
+ }
+ xidlist = *xidlist.next;
+ }
+
+ /* Free all pallocs made by TwoPhaseGetXidByVXid() */
+ while (xidlist_ptr)
+ {
+ void *chunk = xidlist_ptr;
+ xidlist_ptr = xidlist_ptr->next;
+ pfree(chunk);
+ }
+ return result;
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4542,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return WaitXact(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4567,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return WaitXact(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4581,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return WaitXact(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4638,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return WaitXact(vxid, xid, wait);
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..ba0ac17359 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -25,6 +25,17 @@
*/
typedef struct GlobalTransactionData *GlobalTransaction;
+/*
+ * XidListEntry is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+
/* GUC variable */
extern PGDLLIMPORT int max_prepared_xacts;
@@ -58,4 +69,6 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v10-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcac.patch (application/octet-stream, 2.2K) download
From 1492b66eb4948ca2162afe139ec35f24183b4c68 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 14:40:16 +0500
Subject: [PATCH v10 2/4] PoC fix for race in RelationBuildDesc() and relcache
invalidation
---
src/backend/utils/cache/relcache.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fd05615e76..a3347b793e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1228,12 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ /* inProgress is in fact the stack, we can safely remove it's top */
+ inProgress[in_progress_offset].relid = InvalidOid;
+ Assert(inProgress[in_progress_offset + 1].relid == InvalidOid);
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2802,6 +2823,13 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
--
2.24.3 (Apple Git-128)
=
v10-0001-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 7.0K) download
From 6edccb3906472e88b57f7642d496c42078fddc81 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v10 1/4] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/t/002_cic_2pc.pl | 238 +++++++++++++++++++++++++++++++
2 files changed, 239 insertions(+), 1 deletion(-)
create mode 100644 contrib/amcheck/t/002_cic_2pc.pl
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefile
index b82f221e50..5b5427d193 100644
--- a/contrib/amcheck/Makefile
+++ b/contrib/amcheck/Makefile
@@ -12,7 +12,7 @@ PGFILEDESC = "amcheck - function for verifying relation integrity"
REGRESS = check check_btree check_heap
-TAP_TESTS = 1
+TAP_TESTS = 2
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/amcheck/t/002_cic_2pc.pl b/contrib/amcheck/t/002_cic_2pc.pl
new file mode 100644
index 0000000000..541ac2e8a9
--- /dev/null
+++ b/contrib/amcheck/t/002_cic_2pc.pl
@@ -0,0 +1,238 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for point-in-time-recovery (PITR) with prepared transactions
+use strict;
+use warnings;
+
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = get_new_node('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(5);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(5);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+$main_h->pump_nb;
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = background_pgbench('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --time=1');
+
+# Run pgbench.
+pgbench(
+ '--no-vacuum --client=5 --time=1',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ \set txid random(1, 1000000000)
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'tx:txid';
+ COMMIT PREPARED 'tx:txid';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = $pgbench_h->result(0); # Need extra check for ($Config{osname} eq "MSWin32") ?
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
+
+# invoke pgbench, with parameters:
+# $opts: options as a string to be split on spaces
+# $stat: expected exit status
+# $out: reference to a regexp list that must match stdout
+# $err: reference to a regexp list that must match stderr
+# $name: name of test for error messages
+# $files: reference to filename/contents dictionary
+# @args: further raw options or arguments
+sub pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my ($opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $node->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+
+# invoke pgbench in background alike background_psql
+sub background_pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my ($dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $node->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $node->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-07 22:19 ` Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-08-07 22:19 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Sun, Aug 08, 2021 at 12:00:55AM +0500, Andrey Borodin wrote:
> Changes:
> 1. Added assert in step 2 (fix for missed invalidation message). I wonder how deep possibly could be RelationBuildDesc() inside RelationBuildDesc() inside RelationBuildDesc() ... ? If the depth is unlimited we, probably, need a better data structure.
I don't know either, hence that quick data structure to delay the question.
debug_discard_caches=3 may help answer the question. RelationBuildDesc()
reads pg_constraint, which is !rd_isnailed. Hence, I expect one can at least
get RelationBuildDesc("pg_constraint") inside RelationBuildDesc("user_table").
> Tomorrow I'll try to cleanup step 1 (tap tests).
For the final commits, I would like to have two commits. The first will fix
the non-2PC bug and add the test for that fix. The second will fix the 2PC
bug and add the additional test. Feel free to structure your changes into
more patch files, so long as it's straightforward to squash them into two
commits that way.
A few things I noticed earlier:
> --- a/contrib/amcheck/Makefile
> +++ b/contrib/amcheck/Makefile
> @@ -12,7 +12,7 @@ PGFILEDESC = "amcheck - function for verifying relation integrity"
>
> REGRESS = check check_btree check_heap
>
> -TAP_TESTS = 1
> +TAP_TESTS = 2
TAP_TESTS is a Boolean flag, not a count. Leave it set to 1.
> --- /dev/null
> +++ b/contrib/amcheck/t/002_cic_2pc.pl
> +$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
> +my $main_timer = IPC::Run::timeout(5);
> +my $reindex_timer = IPC::Run::timeout(5);
Values this short cause spurious buildfarm failures. Generally, use 180s for
timeouts that do not expire in a successful run.
> +# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
> +# CIC will deadlock with itself occasionally.
Consider fixing that by taking an advisory lock before running CIC. However,
if use of separate pgbench executions works better, feel free to keep that.
> +pgbench(
> + '--no-vacuum --client=5 --time=1',
Consider using a transaction count instead of --time. That way, slow machines
still catch the bug, and fast machines waste less time. For the "concurrent
OID generation" test, I tuned the transaction count so the test failed about
half the time[1] on a buggy build.
> + \set txid random(1, 1000000000)
> + BEGIN;
> + INSERT INTO tbl VALUES(0);
> + PREPARE TRANSACTION 'tx:txid';
Try "PREPARE TRANSACTION 'c:client_id'" (or c:client_id:txid) to eliminate the
chance of collision.
[1] https://postgr.es/m/20160215171129.GA347322@tornado.leadboat.com
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-08-08 11:31 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-08 11:31 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
> 8 авг. 2021 г., в 03:19, Noah Misch <noah@leadboat.com> написал(а):
>
> On Sun, Aug 08, 2021 at 12:00:55AM +0500, Andrey Borodin wrote:
>> Changes:
>> 1. Added assert in step 2 (fix for missed invalidation message). I wonder how deep possibly could be RelationBuildDesc() inside RelationBuildDesc() inside RelationBuildDesc() ... ? If the depth is unlimited we, probably, need a better data structure.
>
> I don't know either, hence that quick data structure to delay the question.
> debug_discard_caches=3 may help answer the question. RelationBuildDesc()
> reads pg_constraint, which is !rd_isnailed. Hence, I expect one can at least
> get RelationBuildDesc("pg_constraint") inside RelationBuildDesc("user_table").
I've toyed around with
$node->append_conf('postgresql.conf', 'debug_invalidate_system_caches_always = 3'); in test.
I had failures only at most with
Assert(in_progress_offset < 4);
See [0] for stack trace. But I do not think that it proves that deeper calls are impossible with other DB schemas.
>> Tomorrow I'll try to cleanup step 1 (tap tests).
>
> For the final commits, I would like to have two commits. The first will fix
> the non-2PC bug and add the test for that fix. The second will fix the 2PC
> bug and add the additional test. Feel free to structure your changes into
> more patch files, so long as it's straightforward to squash them into two
> commits that way.
Done so.
Step 1. Test for CIC with regular transactions.
Step 2. Fix
Step 3. Test for CIC with 2PC
Step 4. Part of the fix that I'm sure about
Step 5. Dubious part of fix...
I had to refactor subs pgbench and pgbench_background to be reused among tests. Though I did not refactor pgbench tests to use this functions: they have two different versions for sub pgbench.
> A few things I noticed earlier:
>
>> --- a/contrib/amcheck/Makefile
>> +++ b/contrib/amcheck/Makefile
>> @@ -12,7 +12,7 @@ PGFILEDESC = "amcheck - function for verifying relation integrity"
>>
>> REGRESS = check check_btree check_heap
>>
>> -TAP_TESTS = 1
>> +TAP_TESTS = 2
>
> TAP_TESTS is a Boolean flag, not a count. Leave it set to 1.
Fixed.
>> --- /dev/null
>> +++ b/contrib/amcheck/t/002_cic_2pc.pl
>
>> +$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
>
>> +my $main_timer = IPC::Run::timeout(5);
>
>> +my $reindex_timer = IPC::Run::timeout(5);
>
> Values this short cause spurious buildfarm failures. Generally, use 180s for
> timeouts that do not expire in a successful run.
Fixed.
>> +# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
>> +# CIC will deadlock with itself occasionally.
>
> Consider fixing that by taking an advisory lock before running CIC. However,
> if use of separate pgbench executions works better, feel free to keep that.
I've tried this approach. Advisory lock owns vxid, thus deadlocking with CIC.
>> +pgbench(
>> + '--no-vacuum --client=5 --time=1',
>
> Consider using a transaction count instead of --time. That way, slow machines
> still catch the bug, and fast machines waste less time. For the "concurrent
> OID generation" test, I tuned the transaction count so the test failed about
> half the time[1] on a buggy build.
I've tuned tests for my laptop to observe probabilistic test to pass sometimes.
>> + \set txid random(1, 1000000000)
>> + BEGIN;
>> + INSERT INTO tbl VALUES(0);
>> + PREPARE TRANSACTION 'tx:txid';
>
> Try "PREPARE TRANSACTION 'c:client_id'" (or c:client_id:txid) to eliminate the
> chance of collision.
Fixed.
How to you think, do we have a chance to fix things before next release on August 12th?
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (5 files, .tar.gz)
v11-0001-Introduce-TAP-test-for-CIC.patch (application/octet-stream, 5.5K) download
From 4ab778c3583d271a3a33b01ec3d098345d6e0947 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v11 1/5] Introduce TAP test for CIC
Implement some pgbench probablistic testing to stress out CREATE INDEX CONCURRENTLY.
---
contrib/amcheck/t/002_cic.pl | 76 ++++++++++++++++++++
src/test/perl/PostgresNode.pm | 126 ++++++++++++++++++++++++++++++++++
2 files changed, 202 insertions(+)
create mode 100644 contrib/amcheck/t/002_cic.pl
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
new file mode 100644
index 0000000000..c920464d09
--- /dev/null
+++ b/contrib/amcheck/t/002_cic.pl
@@ -0,0 +1,76 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 4;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = get_new_node('CIC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+#
+# Stress CIC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=200');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=200',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent transactions',
+ {
+ '002_pgbench_concurrent_transaction' =>
+ q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 2027cbf43d..6aa524f4bc 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1875,6 +1875,132 @@ sub background_psql
=pod
+=item $node->pgbench($opts, $stat, $out, $err, $name, $files, @args)
+
+Invoke pgbench, with parameters and files.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $stat
+
+Expected exit status.
+
+=item $out
+
+Reference to a regexp list that must match stdout.
+
+=item $err
+
+Reference to a regexp list that must match stderr.
+
+=item $name
+
+Name of test for error messages.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=item @args
+
+Further raw options or arguments.
+
+=back
+
+=cut
+sub pgbench
+{
+ my ($self, $opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $self->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+=pod
+
+=item $node->background_pgbench($dbname, \$stdin, \$stdout, $timer, $files, %opts) => harness
+
+Invoke B<pgbench> in background alike background_psql.
+
+=cut
+sub background_pgbench
+{
+ my ($self, $dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $self->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
+=pod
+
=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
--
2.24.3 (Apple Git-128)
=
v11-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcac.patch (application/octet-stream, 2.2K) download
From a917cae8fe1730fcf715aee2dabd1dd9d32e15b6 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 14:40:16 +0500
Subject: [PATCH v11 2/5] PoC fix for race in RelationBuildDesc() and relcache
invalidation
---
src/backend/utils/cache/relcache.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fd05615e76..a3347b793e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1228,12 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ /* inProgress is in fact the stack, we can safely remove it's top */
+ inProgress[in_progress_offset].relid = InvalidOid;
+ Assert(inProgress[in_progress_offset + 1].relid == InvalidOid);
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2802,6 +2823,13 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
--
2.24.3 (Apple Git-128)
=
v11-0003-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 4.2K) download
From 07357204b8283b70c82e6144758b24f9c01f2387 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 8 Aug 2021 14:46:50 +0500
Subject: [PATCH v11 3/5] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/t/003_cic_2pc.pl | 146 +++++++++++++++++++++++++++++++
1 file changed, 146 insertions(+)
create mode 100644 contrib/amcheck/t/003_cic_2pc.pl
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
new file mode 100644
index 0000000000..b11cd76975
--- /dev/null
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -0,0 +1,146 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY with prepared transactions
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = get_new_node('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(180);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(180);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+$main_h->pump_nb;
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=100');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=100',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v11-0004-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxi.patch (application/octet-stream, 8.5K) download
From 38ae6f779b7cee9f428633db49508b90ad289c3b Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v11 4/5] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2pc
---
src/backend/access/transam/twophase.c | 47 ++++++++++-
src/backend/storage/lmgr/lock.c | 115 ++++++++++++++++++++++----
src/include/access/twophase.h | 13 +++
3 files changed, 154 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index f67d813c56..ef551dd0d3 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +847,46 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+XidListEntry
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
+ VirtualTransactionId proc_vxid;
+ GET_VXID_FROM_PGPROC(vxid, *proc);
+
+ if (VirtualTransactionIdEquals(vxid, proc_vxid))
+ {
+ if (result.xid != InvalidTransactionId)
+ {
+ /* Already has candidate - need to alloc some space */
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 108b4d9023..137eeedb93 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -386,6 +386,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -3017,6 +3018,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
+ /* Prefer real Xid over local VXid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxid.backendId = InvalidBackendId;
+ vxid.localTransactionId = proc->xid;
+ }
if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3078,6 +3085,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
+ /* Prefer real Xid over local VXid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxid.backendId = InvalidBackendId;
+ vxid.localTransactionId = proc->xid;
+ }
if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -4447,6 +4460,76 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * WaitXact
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * two-phase procarray entries.
+ */
+static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ XidListEntry xidlist;
+ XidListEntry *xidlist_ptr = NULL; /* pointer to TwoPhaseGetXidByVXid()s pallocs */
+ bool result;
+
+ if (TransactionIdIsValid(xid))
+ {
+ /* We know exact xid - no need to search in 2PC state */
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /* You cant have vxid among 2PCs if you have no 2PCs */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ /*
+ * We must search for vxids in 2pc state
+ * XXX: O(N*N) complexity where N is number of prepared xacts
+ */
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ /* Return if transaction is gone entirely */
+ if (!TransactionIdIsValid(xidlist.xid))
+ return true;
+ xidlist_ptr = xidlist.next;
+ }
+
+
+ while (true)
+ {
+ /* Iterate over possible xids */
+ Assert(TransactionIdIsValid(xidlist.xid));
+ SET_LOCKTAG_TRANSACTION(tag, xidlist.xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ {
+ result = false;
+ break;
+ }
+
+ LockRelease(&tag, ShareLock, false);
+
+ if (xidlist.next == NULL)
+ {
+ result = true;
+ break;
+ }
+ xidlist = *xidlist.next;
+ }
+
+ /* Free all pallocs made by TwoPhaseGetXidByVXid() */
+ while (xidlist_ptr)
+ {
+ void *chunk = xidlist_ptr;
+ xidlist_ptr = xidlist_ptr->next;
+ pfree(chunk);
+ }
+ return result;
+}
+
/*
* VirtualXactLock
*
@@ -4459,25 +4542,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return WaitXact(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4491,7 +4567,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return WaitXact(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4505,9 +4581,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return WaitXact(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4559,7 +4638,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return WaitXact(vxid, xid, wait);
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index 91786da784..ba0ac17359 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -25,6 +25,17 @@
*/
typedef struct GlobalTransactionData *GlobalTransaction;
+/*
+ * XidListEntry is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+
/* GUC variable */
extern PGDLLIMPORT int max_prepared_xacts;
@@ -58,4 +69,6 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v11-0005-PoC-fix-clear-xid.patch (application/octet-stream, 1.0K) download
From 70a837ece8afe1c1487cb802c3d0df36b3cf5175 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 7 Aug 2021 20:27:14 +0500
Subject: [PATCH v11 5/5] PoC fix clear xid
---
src/backend/access/transam/xact.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 441445927e..b24a3de8d4 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2500,7 +2500,6 @@ PrepareTransaction(void)
* done *after* the prepared transaction has been marked valid, else
* someone may think it is unlocked and recyclable.
*/
- ProcArrayClearTransaction(MyProc);
/*
* In normal commit-processing, this is all non-critical post-transaction
@@ -2535,6 +2534,8 @@ PrepareTransaction(void)
PostPrepare_MultiXact(xid);
PostPrepare_Locks(xid);
+
+ ProcArrayClearTransaction(MyProc);
PostPrepare_PredicateLocks(xid);
ResourceOwnerRelease(TopTransactionResourceOwner,
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-08 16:37 ` Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-08-08 16:37 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Sun, Aug 08, 2021 at 04:31:07PM +0500, Andrey Borodin wrote:
> > 8 авг. 2021 г., в 03:19, Noah Misch <noah@leadboat.com> написал(а):
> > On Sun, Aug 08, 2021 at 12:00:55AM +0500, Andrey Borodin wrote:
> >> Changes:
> >> 1. Added assert in step 2 (fix for missed invalidation message). I wonder how deep possibly could be RelationBuildDesc() inside RelationBuildDesc() inside RelationBuildDesc() ... ? If the depth is unlimited we, probably, need a better data structure.
> >
> > I don't know either, hence that quick data structure to delay the question.
> > debug_discard_caches=3 may help answer the question. RelationBuildDesc()
> > reads pg_constraint, which is !rd_isnailed. Hence, I expect one can at least
> > get RelationBuildDesc("pg_constraint") inside RelationBuildDesc("user_table").
> I've toyed around with
> $node->append_conf('postgresql.conf', 'debug_invalidate_system_caches_always = 3'); in test.
> I had failures only at most with
> Assert(in_progress_offset < 4);
> See [0] for stack trace. But I do not think that it proves that deeper calls are impossible with other DB schemas.
I didn't find the [0] link in your message; can you send it again? I suspect
no rel can appear more than once in the stack, and all but one rel will be a
system catalog. That said, dynamically growing the array is reasonable, even
if a small maximum depth theoretically exists.
> Step 1. Test for CIC with regular transactions.
> Step 2. Fix
> Step 3. Test for CIC with 2PC
> Step 4. Part of the fix that I'm sure about
> Step 5. Dubious part of fix...
> How to you think, do we have a chance to fix things before next release on August 12th?
No. At a minimum, we still need to convince ourselves that step 5 is correct.
Even if that were already done, commits to released branches get more cautious
in the few days before the wrap date (tomorrow). Once it's committed, you
could contact pgsql-release@postgresql.org to propose an out-of-cycle release.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-08-14 22:13 ` Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-08-14 22:13 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Sun, Aug 08, 2021 at 09:37:52AM -0700, Noah Misch wrote:
> On Sun, Aug 08, 2021 at 04:31:07PM +0500, Andrey Borodin wrote:
> > Step 1. Test for CIC with regular transactions.
> > Step 2. Fix
> > Step 3. Test for CIC with 2PC
> > Step 4. Part of the fix that I'm sure about
> > Step 5. Dubious part of fix...
When I applied all five to commit b33259e^ (shortly before the PostgresNode
compatibility break) and ran your tests in a loop, I got more 2PC "lacks
matching index tuple" within one minute. Here's how I ran the tests:
make -j20 && while make -C contrib/amcheck check REGRESS= PROVE_FLAGS=--timer PROVE_TESTS='t/002_cic.pl t/003_cic_2pc.pl' NO_TEMP_INSTALL=1; do date; done
Do you see failures with that loop? If so, can you diagnose them? (So far,
I've not seen a failure from the 1PC test.)
On Thu, Jul 29, 2021 at 10:25:48PM -0400, Noah Misch wrote:
> I haven't checked whether other inval message types have the same hazard.
I will look into this next. (It probably doesn't explain 2PC failures.)
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-08-15 08:45 ` Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-08-15 08:45 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Sat, Aug 14, 2021 at 03:13:28PM -0700, Noah Misch wrote:
> On Sun, Aug 08, 2021 at 09:37:52AM -0700, Noah Misch wrote:
> > On Sun, Aug 08, 2021 at 04:31:07PM +0500, Andrey Borodin wrote:
> > > Step 1. Test for CIC with regular transactions.
> > > Step 2. Fix
> > > Step 3. Test for CIC with 2PC
> > > Step 4. Part of the fix that I'm sure about
> > > Step 5. Dubious part of fix...
>
> When I applied all five to commit b33259e^ (shortly before the PostgresNode
> compatibility break) and ran your tests in a loop, I got more 2PC "lacks
> matching index tuple" within one minute. Here's how I ran the tests:
>
> make -j20 && while make -C contrib/amcheck check REGRESS= PROVE_FLAGS=--timer PROVE_TESTS='t/002_cic.pl t/003_cic_2pc.pl' NO_TEMP_INSTALL=1; do date; done
>
> Do you see failures with that loop? If so, can you diagnose them?
See below.
> (So far, I've not seen a failure from the 1PC test.)
I eventually did see one. Just one 1PC failure in six hours of 1PC test
runtime, though.
> On Thu, Jul 29, 2021 at 10:25:48PM -0400, Noah Misch wrote:
> > I haven't checked whether other inval message types have the same hazard.
>
> I will look into this next. (It probably doesn't explain 2PC failures.)
Now that I've looked, other inval hazards might indeed explain the remaining
2PC and 1PC failures. I'll plug those holes and see what failures, if any,
continue to happen.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-08-15 11:09 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-16 05:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 2 replies; 107+ messages in thread
From: Andrey Borodin @ 2021-08-15 11:09 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
> 15 авг. 2021 г., в 13:45, Noah Misch <noah@leadboat.com> написал(а):
>>
>> Do you see failures with that loop? If so, can you diagnose them?
>
> See below.
I do not observe failure on my laptop, though reproduced it on a linux server.
I've fixed one bug in TwoPhaseGetXidByVXid(). Also rebased on actual master.
>
>> (So far, I've not seen a failure from the 1PC test.)
>
> I eventually did see one. Just one 1PC failure in six hours of 1PC test
> runtime, though.
I've attached a patch that reproduces the problem in 30sec on my server.
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (6 files, .tar.gz)
v12-0006-Do-CIC-test-time-based-to-ensure-bug-repro.patch (application/octet-stream, 925B) download
From 01e82d19be928947d359b449ad6bb6df03551491 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 15 Aug 2021 11:04:35 +0000
Subject: [PATCH v12 6/6] Do CIC test time-based to ensure bug repro
---
contrib/amcheck/t/002_cic.pl | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
index db5d44846f..d11bda57f4 100644
--- a/contrib/amcheck/t/002_cic.pl
+++ b/contrib/amcheck/t/002_cic.pl
@@ -44,11 +44,11 @@ my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_ou
SELECT bt_index_check('idx',true);
)
},
- '--no-vacuum --client=1 --transactions=200');
+ '--no-vacuum --client=1 --time=30');
# Run pgbench.
$node->pgbench(
- '--no-vacuum --client=5 --transactions=200',
+ '--no-vacuum --client=5 --time=30',
0,
[qr{actually processed}],
[qr{^$}],
--
2.24.3 (Apple Git-128)
=
v12-0005-PoC-fix-clear-xid.patch (application/octet-stream, 1.0K) download
From 7e47dae2828d88ddb2161fda0c3b08a158c6cf37 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 7 Aug 2021 20:27:14 +0500
Subject: [PATCH v12 5/6] PoC fix clear xid
---
src/backend/access/transam/xact.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 387f80419a..9b19d939eb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2500,7 +2500,6 @@ PrepareTransaction(void)
* done *after* the prepared transaction has been marked valid, else
* someone may think it is unlocked and recyclable.
*/
- ProcArrayClearTransaction(MyProc);
/*
* In normal commit-processing, this is all non-critical post-transaction
@@ -2535,6 +2534,8 @@ PrepareTransaction(void)
PostPrepare_MultiXact(xid);
PostPrepare_Locks(xid);
+
+ ProcArrayClearTransaction(MyProc);
PostPrepare_PredicateLocks(xid);
ResourceOwnerRelease(TopTransactionResourceOwner,
--
2.24.3 (Apple Git-128)
=
v12-0004-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxi.patch (application/octet-stream, 8.6K) download
From 6db9cafd146db1a645bb6885157b0e1f032765e0 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v12 4/6] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2pc
---
src/backend/access/transam/twophase.c | 47 ++++++++++-
src/backend/storage/lmgr/lock.c | 115 ++++++++++++++++++++++----
src/include/access/twophase.h | 14 ++++
3 files changed, 155 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 2156de187c..4823ded52b 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +847,46 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+XidListEntry
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
+ VirtualTransactionId proc_vxid;
+ GET_VXID_FROM_PGPROC(proc_vxid, *proc);
+
+ if (VirtualTransactionIdEquals(vxid, proc_vxid))
+ {
+ if (result.xid != InvalidTransactionId)
+ {
+ /* Already has candidate - need to alloc some space */
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..5d433d932d 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -387,6 +387,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -3018,6 +3019,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
/* Conflict! */
GET_VXID_FROM_PGPROC(vxid, *proc);
+ /* Prefer real Xid over local VXid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxid.backendId = InvalidBackendId;
+ vxid.localTransactionId = proc->xid;
+ }
if (VirtualTransactionIdIsValid(vxid))
vxids[count++] = vxid;
/* else, xact already committed or aborted */
@@ -3079,6 +3086,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
GET_VXID_FROM_PGPROC(vxid, *proc);
+ /* Prefer real Xid over local VXid */
+ if (TransactionIdIsValid(proc->xid))
+ {
+ vxid.backendId = InvalidBackendId;
+ vxid.localTransactionId = proc->xid;
+ }
if (VirtualTransactionIdIsValid(vxid))
{
int i;
@@ -4529,6 +4542,76 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * WaitXact
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * two-phase procarray entries.
+ */
+static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ XidListEntry xidlist;
+ XidListEntry *xidlist_ptr = NULL; /* pointer to TwoPhaseGetXidByVXid()s pallocs */
+ bool result;
+
+ if (TransactionIdIsValid(xid))
+ {
+ /* We know exact xid - no need to search in 2PC state */
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /* You cant have vxid among 2PCs if you have no 2PCs */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ /*
+ * We must search for vxids in 2pc state
+ * XXX: O(N*N) complexity where N is number of prepared xacts
+ */
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ /* Return if transaction is gone entirely */
+ if (!TransactionIdIsValid(xidlist.xid))
+ return true;
+ xidlist_ptr = xidlist.next;
+ }
+
+
+ while (true)
+ {
+ /* Iterate over possible xids */
+ Assert(TransactionIdIsValid(xidlist.xid));
+ SET_LOCKTAG_TRANSACTION(tag, xidlist.xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ {
+ result = false;
+ break;
+ }
+
+ LockRelease(&tag, ShareLock, false);
+
+ if (xidlist.next == NULL)
+ {
+ result = true;
+ break;
+ }
+ xidlist = *xidlist.next;
+ }
+
+ /* Free all pallocs made by TwoPhaseGetXidByVXid() */
+ while (xidlist_ptr)
+ {
+ void *chunk = xidlist_ptr;
+ xidlist_ptr = xidlist_ptr->next;
+ pfree(chunk);
+ }
+ return result;
+}
+
/*
* VirtualXactLock
*
@@ -4541,25 +4624,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return WaitXact(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4573,7 +4649,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return WaitXact(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4587,9 +4663,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return WaitXact(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4641,7 +4720,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return WaitXact(vxid, xid, wait);
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index e27e1a8fe8..a5f28d3a80 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -25,6 +25,17 @@
*/
typedef struct GlobalTransactionData *GlobalTransaction;
+/*
+ * XidListEntry is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+
/* GUC variable */
extern PGDLLIMPORT int max_prepared_xacts;
@@ -58,6 +69,9 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
extern bool LookupGXact(const char *gid, XLogRecPtr prepare_at_lsn,
TimestampTz origin_prepare_timestamp);
+
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v12-0003-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 4.2K) download
From ad596e84be72f1d0df948d7d9ea3a0490e977e3b Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 8 Aug 2021 14:46:50 +0500
Subject: [PATCH v12 3/6] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/t/003_cic_2pc.pl | 146 +++++++++++++++++++++++++++++++
1 file changed, 146 insertions(+)
create mode 100644 contrib/amcheck/t/003_cic_2pc.pl
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
new file mode 100644
index 0000000000..dc14777509
--- /dev/null
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -0,0 +1,146 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY with prepared transactions
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(180);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(180);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+$main_h->pump_nb;
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+);
+$main_h->pump_nb;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=100');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=100',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v12-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcac.patch (application/octet-stream, 2.2K) download
From 929736512ebf8eb9ac6ddaaf49b9e6148d72cfbb Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 14:40:16 +0500
Subject: [PATCH v12 2/6] PoC fix for race in RelationBuildDesc() and relcache
invalidation
---
src/backend/utils/cache/relcache.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 13d9994af3..7eec7b1f41 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1228,12 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ /* inProgress is in fact the stack, we can safely remove it's top */
+ inProgress[in_progress_offset].relid = InvalidOid;
+ Assert(inProgress[in_progress_offset + 1].relid == InvalidOid);
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2805,6 +2826,13 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
--
2.24.3 (Apple Git-128)
=
v12-0001-Introduce-TAP-test-for-CIC.patch (application/octet-stream, 5.5K) download
From 0002090a069a78374b316580830682f3350dd9d8 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v12 1/6] Introduce TAP test for CIC
Implement some pgbench probablistic testing to stress out CREATE INDEX CONCURRENTLY.
---
contrib/amcheck/t/002_cic.pl | 76 ++++++++++++++++++++
src/test/perl/PostgresNode.pm | 126 ++++++++++++++++++++++++++++++++++
2 files changed, 202 insertions(+)
create mode 100644 contrib/amcheck/t/002_cic.pl
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
new file mode 100644
index 0000000000..db5d44846f
--- /dev/null
+++ b/contrib/amcheck/t/002_cic.pl
@@ -0,0 +1,76 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 4;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+#
+# Stress CIC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=200');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=200',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent transactions',
+ {
+ '002_pgbench_concurrent_transaction' =>
+ q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8158ea5b2f..631eb5618e 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1912,6 +1912,132 @@ sub background_psql
=pod
+=item $node->pgbench($opts, $stat, $out, $err, $name, $files, @args)
+
+Invoke pgbench, with parameters and files.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $stat
+
+Expected exit status.
+
+=item $out
+
+Reference to a regexp list that must match stdout.
+
+=item $err
+
+Reference to a regexp list that must match stderr.
+
+=item $name
+
+Name of test for error messages.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=item @args
+
+Further raw options or arguments.
+
+=back
+
+=cut
+sub pgbench
+{
+ my ($self, $opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $self->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+=pod
+
+=item $node->background_pgbench($dbname, \$stdin, \$stdout, $timer, $files, %opts) => harness
+
+Invoke B<pgbench> in background alike background_psql.
+
+=cut
+sub background_pgbench
+{
+ my ($self, $dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $self->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
+=pod
+
=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-15 13:45 ` Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 107+ messages in thread
From: Andres Freund @ 2021-08-15 13:45 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
Hi,
On 2021-08-15 16:09:37 +0500, Andrey Borodin wrote:
> From 929736512ebf8eb9ac6ddaaf49b9e6148d72cfbb Mon Sep 17 00:00:00 2001
> From: Andrey Borodin <amborodin@acm.org>
> Date: Fri, 30 Jul 2021 14:40:16 +0500
> Subject: [PATCH v12 2/6] PoC fix for race in RelationBuildDesc() and relcache
> invalidation
>
> ---
> src/backend/utils/cache/relcache.c | 28 ++++++++++++++++++++++++++++
> 1 file changed, 28 insertions(+)
>
> diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
> index 13d9994af3..7eec7b1f41 100644
> --- a/src/backend/utils/cache/relcache.c
> +++ b/src/backend/utils/cache/relcache.c
> @@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
> * (suggesting we are trying to access a just-deleted relation).
> * Any other error is reported via elog.
> */
> +typedef struct InProgressRels {
> + Oid relid;
> + bool invalidated;
> +} InProgressRels;
> +static InProgressRels inProgress[100];
> +
> static Relation
> RelationBuildDesc(Oid targetRelId, bool insertIt)
> {
> + int in_progress_offset;
> Relation relation;
> Oid relid;
> HeapTuple pg_class_tuple;
> @@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
> }
> #endif
>
> + for (in_progress_offset = 0;
> + OidIsValid(inProgress[in_progress_offset].relid);
> + in_progress_offset++)
> + ;
> + inProgress[in_progress_offset].relid = targetRelId;
> +retry:
> + inProgress[in_progress_offset].invalidated = false;
> +
> /*
> * find the tuple in pg_class corresponding to the given relation id
> */
> @@ -1213,6 +1228,12 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
> */
> heap_freetuple(pg_class_tuple);
>
> + if (inProgress[in_progress_offset].invalidated)
> + goto retry; /* TODO free old one */
> + /* inProgress is in fact the stack, we can safely remove it's top */
> + inProgress[in_progress_offset].relid = InvalidOid;
> + Assert(inProgress[in_progress_offset + 1].relid == InvalidOid);
> +
> /*
> * Insert newly created relation into relcache hash table, if requested.
> *
> @@ -2805,6 +2826,13 @@ RelationCacheInvalidateEntry(Oid relationId)
> relcacheInvalsReceived++;
> RelationFlushRelation(relation);
> }
> + else
> + {
> + int i;
> + for (i = 0; OidIsValid(inProgress[i].relid); i++)
> + if (inProgress[i].relid == relationId)
> + inProgress[i].invalidated = true;
> + }
> }
Desperately needs comments. Without a commit message and without
comments it's hard to review this without re-reading the entire thread -
which approximately nobody will do.
> From 7e47dae2828d88ddb2161fda0c3b08a158c6cf37 Mon Sep 17 00:00:00 2001
> From: Andrey Borodin <amborodin@acm.org>
> Date: Sat, 7 Aug 2021 20:27:14 +0500
> Subject: [PATCH v12 5/6] PoC fix clear xid
>
> ---
> src/backend/access/transam/xact.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
> index 387f80419a..9b19d939eb 100644
> --- a/src/backend/access/transam/xact.c
> +++ b/src/backend/access/transam/xact.c
> @@ -2500,7 +2500,6 @@ PrepareTransaction(void)
> * done *after* the prepared transaction has been marked valid, else
> * someone may think it is unlocked and recyclable.
> */
> - ProcArrayClearTransaction(MyProc);
>
> /*
> * In normal commit-processing, this is all non-critical post-transaction
> @@ -2535,6 +2534,8 @@ PrepareTransaction(void)
> PostPrepare_MultiXact(xid);
>
> PostPrepare_Locks(xid);
> +
> + ProcArrayClearTransaction(MyProc);
> PostPrepare_PredicateLocks(xid);
The comment above ProcArrayClearTransaction would need to be moved and
updated...
> From 6db9cafd146db1a645bb6885157b0e1f032765e0 Mon Sep 17 00:00:00 2001
> From: Andrey Borodin <amborodin@acm.org>
> Date: Mon, 19 Jul 2021 11:50:02 +0500
> Subject: [PATCH v12 4/6] Fix CREATE INDEX CONCURRENTLY in precence of vxids
> converted to 2pc
...
> +/*
> + * TwoPhaseGetXidByVXid
> + * Try to lookup for vxid among prepared xacts
> + */
> +XidListEntry
> +TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
> +{
> + int i;
> + XidListEntry result;
> + result.next = NULL;
> + result.xid = InvalidTransactionId;
> +
> + LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
> +
> + for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
> + {
> + GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
> + PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
> + VirtualTransactionId proc_vxid;
> + GET_VXID_FROM_PGPROC(proc_vxid, *proc);
> +
> + if (VirtualTransactionIdEquals(vxid, proc_vxid))
> + {
> + if (result.xid != InvalidTransactionId)
> + {
> + /* Already has candidate - need to alloc some space */
> + XidListEntry *copy = palloc(sizeof(XidListEntry));
> + copy->next = result.next;
> + copy->xid = result.xid;
> + result.next = copy;
> + }
> + result.xid = gxact->xid;
> + }
> + }
> +
> + LWLockRelease(TwoPhaseStateLock);
> +
> + return result;
> +}
Dynamic memory allocations while holding a fairly central lock - one
which is now going to be more contended - doesn't seem great.
Is the memory context this is called in guaranteed to be of a proper
duration? Including being reset in case there's an error at some point
before the memory is freed?
> +/*
> + * WaitXact
> + *
> + * Wait for xid completition if have xid. Otherwise try to find xid among
> + * two-phase procarray entries.
> + */
> +static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait)
> +{
> + LockAcquireResult lar;
> + LOCKTAG tag;
> + XidListEntry xidlist;
> + XidListEntry *xidlist_ptr = NULL; /* pointer to TwoPhaseGetXidByVXid()s pallocs */
> + bool result;
> +
> + if (TransactionIdIsValid(xid))
> + {
> + /* We know exact xid - no need to search in 2PC state */
> + xidlist.xid = xid;
> + xidlist.next = NULL;
> + }
> + else
> + {
> + /* You cant have vxid among 2PCs if you have no 2PCs */
> + if (max_prepared_xacts == 0)
> + return true;
> +
> + /*
> + * We must search for vxids in 2pc state
> + * XXX: O(N*N) complexity where N is number of prepared xacts
> + */
> + xidlist = TwoPhaseGetXidByVXid(vxid);
> + /* Return if transaction is gone entirely */
> + if (!TransactionIdIsValid(xidlist.xid))
> + return true;
> + xidlist_ptr = xidlist.next;
> + }
Perhaps I missed this - but won't we constantly enter this path for
non-2pc transactions? E.g.
> @@ -4573,7 +4649,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
> */
> proc = BackendIdGetProc(vxid.backendId);
> if (proc == NULL)
> - return true;
> + return WaitXact(vxid, InvalidTransactionId, wait);
>
> /*
> * We must acquire this lock before checking the backendId and lxid
> @@ -4587,9 +4663,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
> || proc->fpLocalTransactionId != vxid.localTransactionId)
> {
> LWLockRelease(&proc->fpInfoLock);
> - return true;
> + return WaitXact(vxid, InvalidTransactionId, wait);
> }
It seems like it's going to add a substantial amount of work even when
no 2PC xacts are involved?
> diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
> index e27e1a8fe8..a5f28d3a80 100644
> --- a/src/include/access/twophase.h
> +++ b/src/include/access/twophase.h
> @@ -25,6 +25,17 @@
> */
> typedef struct GlobalTransactionData *GlobalTransaction;
>
> +/*
> + * XidListEntry is expected to be used as list very rarely. Under normal
> + * circumstances TwoPhaseGetXidByVXid() returns only one xid.
> + * But under certain conditions can return many xids or nothing.
> + */
> +typedef struct XidListEntry
> +{
> + TransactionId xid;
> + struct XidListEntry* next;
> +} XidListEntry;
I don't think we should invent additional ad-hoc list types.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
@ 2021-08-29 07:27 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-29 07:27 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
Hi!
> 15 авг. 2021 г., в 18:45, Andres Freund <andres@anarazel.de> написал(а):
>
> Hi,
>
> On 2021-08-15 16:09:37 +0500, Andrey Borodin wrote:
>> From 929736512ebf8eb9ac6ddaaf49b9e6148d72cfbb Mon Sep 17 00:00:00 2001
>> From: Andrey Borodin <amborodin@acm.org>
>> Date: Fri, 30 Jul 2021 14:40:16 +0500
>> Subject: [PATCH v12 2/6] PoC fix for race in RelationBuildDesc() and relcache
>> invalidation
>>
>> ---
>> src/backend/utils/cache/relcache.c | 28 ++++++++++++++++++++++++++++
>> 1 file changed, 28 insertions(+)
>>
>> diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
>> index 13d9994af3..7eec7b1f41 100644
>> --- a/src/backend/utils/cache/relcache.c
>> +++ b/src/backend/utils/cache/relcache.c
>> @@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
>> * (suggesting we are trying to access a just-deleted relation).
>> * Any other error is reported via elog.
>> */
>> +typedef struct InProgressRels {
>> + Oid relid;
>> + bool invalidated;
>> +} InProgressRels;
>> +static InProgressRels inProgress[100];
>> +
>> static Relation
>> RelationBuildDesc(Oid targetRelId, bool insertIt)
>> {
>> + int in_progress_offset;
>> Relation relation;
>> Oid relid;
>> HeapTuple pg_class_tuple;
>> @@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
>> }
>> #endif
>>
>> + for (in_progress_offset = 0;
>> + OidIsValid(inProgress[in_progress_offset].relid);
>> + in_progress_offset++)
>> + ;
>> + inProgress[in_progress_offset].relid = targetRelId;
>> +retry:
>> + inProgress[in_progress_offset].invalidated = false;
>> +
>> /*
>> * find the tuple in pg_class corresponding to the given relation id
>> */
>> @@ -1213,6 +1228,12 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
>> */
>> heap_freetuple(pg_class_tuple);
>>
>> + if (inProgress[in_progress_offset].invalidated)
>> + goto retry; /* TODO free old one */
>> + /* inProgress is in fact the stack, we can safely remove it's top */
>> + inProgress[in_progress_offset].relid = InvalidOid;
>> + Assert(inProgress[in_progress_offset + 1].relid == InvalidOid);
>> +
>> /*
>> * Insert newly created relation into relcache hash table, if requested.
>> *
>> @@ -2805,6 +2826,13 @@ RelationCacheInvalidateEntry(Oid relationId)
>> relcacheInvalsReceived++;
>> RelationFlushRelation(relation);
>> }
>> + else
>> + {
>> + int i;
>> + for (i = 0; OidIsValid(inProgress[i].relid); i++)
>> + if (inProgress[i].relid == relationId)
>> + inProgress[i].invalidated = true;
>> + }
>> }
>
> Desperately needs comments. Without a commit message and without
> comments it's hard to review this without re-reading the entire thread -
> which approximately nobody will do.
I've added some comments. But it seems we should use dynamic allocations instead of 100-based array.
>
>
>> From 7e47dae2828d88ddb2161fda0c3b08a158c6cf37 Mon Sep 17 00:00:00 2001
>> From: Andrey Borodin <amborodin@acm.org>
>> Date: Sat, 7 Aug 2021 20:27:14 +0500
>> Subject: [PATCH v12 5/6] PoC fix clear xid
>>
>> ---
>> src/backend/access/transam/xact.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
>> index 387f80419a..9b19d939eb 100644
>> --- a/src/backend/access/transam/xact.c
>> +++ b/src/backend/access/transam/xact.c
>> @@ -2500,7 +2500,6 @@ PrepareTransaction(void)
>> * done *after* the prepared transaction has been marked valid, else
>> * someone may think it is unlocked and recyclable.
>> */
>> - ProcArrayClearTransaction(MyProc);
>>
>> /*
>> * In normal commit-processing, this is all non-critical post-transaction
>> @@ -2535,6 +2534,8 @@ PrepareTransaction(void)
>> PostPrepare_MultiXact(xid);
>>
>> PostPrepare_Locks(xid);
>> +
>> + ProcArrayClearTransaction(MyProc);
>> PostPrepare_PredicateLocks(xid);
>
> The comment above ProcArrayClearTransaction would need to be moved and
> updated...
Fixed.
>
>> From 6db9cafd146db1a645bb6885157b0e1f032765e0 Mon Sep 17 00:00:00 2001
>> From: Andrey Borodin <amborodin@acm.org>
>> Date: Mon, 19 Jul 2021 11:50:02 +0500
>> Subject: [PATCH v12 4/6] Fix CREATE INDEX CONCURRENTLY in precence of vxids
>> converted to 2pc
> ...
>
>> +/*
>> + * TwoPhaseGetXidByVXid
>> + * Try to lookup for vxid among prepared xacts
>> + */
>> +XidListEntry
>> +TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
>> +{
>> + int i;
>> + XidListEntry result;
>> + result.next = NULL;
>> + result.xid = InvalidTransactionId;
>> +
>> + LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
>> +
>> + for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
>> + {
>> + GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
>> + PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
>> + VirtualTransactionId proc_vxid;
>> + GET_VXID_FROM_PGPROC(proc_vxid, *proc);
>> +
>> + if (VirtualTransactionIdEquals(vxid, proc_vxid))
>> + {
>> + if (result.xid != InvalidTransactionId)
>> + {
>> + /* Already has candidate - need to alloc some space */
>> + XidListEntry *copy = palloc(sizeof(XidListEntry));
>> + copy->next = result.next;
>> + copy->xid = result.xid;
>> + result.next = copy;
>> + }
>> + result.xid = gxact->xid;
>> + }
>> + }
>> +
>> + LWLockRelease(TwoPhaseStateLock);
>> +
>> + return result;
>> +}
>
> Dynamic memory allocations while holding a fairly central lock - one
> which is now going to be more contended - doesn't seem great.
>
> Is the memory context this is called in guaranteed to be of a proper
> duration? Including being reset in case there's an error at some point
> before the memory is freed?
I've removed custom list and all memory allocations. If there are multiple 2PCs with same vxid - we just wait for one, then retry.
>> +/*
>> + * WaitXact
>> + *
>> + * Wait for xid completition if have xid. Otherwise try to find xid among
>> + * two-phase procarray entries.
>> + */
>> +static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait)
>> +{
>> + LockAcquireResult lar;
>> + LOCKTAG tag;
>> + XidListEntry xidlist;
>> + XidListEntry *xidlist_ptr = NULL; /* pointer to TwoPhaseGetXidByVXid()s pallocs */
>> + bool result;
>> +
>> + if (TransactionIdIsValid(xid))
>> + {
>> + /* We know exact xid - no need to search in 2PC state */
>> + xidlist.xid = xid;
>> + xidlist.next = NULL;
>> + }
>> + else
>> + {
>> + /* You cant have vxid among 2PCs if you have no 2PCs */
>> + if (max_prepared_xacts == 0)
>> + return true;
>> +
>> + /*
>> + * We must search for vxids in 2pc state
>> + * XXX: O(N*N) complexity where N is number of prepared xacts
>> + */
>> + xidlist = TwoPhaseGetXidByVXid(vxid);
>> + /* Return if transaction is gone entirely */
>> + if (!TransactionIdIsValid(xidlist.xid))
>> + return true;
>> + xidlist_ptr = xidlist.next;
>> + }
>
> Perhaps I missed this - but won't we constantly enter this path for
> non-2pc transactions? E.g.
I've restored heuristics that if it's not 2PC - we just exit from WaitPreparedXact().
>
>> @@ -4573,7 +4649,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
>> */
>> proc = BackendIdGetProc(vxid.backendId);
>> if (proc == NULL)
>> - return true;
>> + return WaitXact(vxid, InvalidTransactionId, wait);
>>
>> /*
>> * We must acquire this lock before checking the backendId and lxid
>> @@ -4587,9 +4663,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
>> || proc->fpLocalTransactionId != vxid.localTransactionId)
>> {
>> LWLockRelease(&proc->fpInfoLock);
>> - return true;
>> + return WaitXact(vxid, InvalidTransactionId, wait);
>> }
>
> It seems like it's going to add a substantial amount of work even when
> no 2PC xacts are involved?
Only if 2PCs are enabled.
>> diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
>> index e27e1a8fe8..a5f28d3a80 100644
>> --- a/src/include/access/twophase.h
>> +++ b/src/include/access/twophase.h
>> @@ -25,6 +25,17 @@
>> */
>> typedef struct GlobalTransactionData *GlobalTransaction;
>>
>> +/*
>> + * XidListEntry is expected to be used as list very rarely. Under normal
>> + * circumstances TwoPhaseGetXidByVXid() returns only one xid.
>> + * But under certain conditions can return many xids or nothing.
>> + */
>> +typedef struct XidListEntry
>> +{
>> + TransactionId xid;
>> + struct XidListEntry* next;
>> +} XidListEntry;
>
> I don't think we should invent additional ad-hoc list types.
Fixed, removed list here entirely.
I'm attaching new version. It works fine on my machines.
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (6 files, .tar.gz)
v13-0001-Introduce-TAP-test-for-CIC.patch (application/octet-stream, 5.6K) download
From c776952049b52b484cf11e2db24233882adc5493 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v13 1/6] Introduce TAP test for CIC
Implement some pgbench probablistic testing to stress out CREATE INDEX CONCURRENTLY.
---
contrib/amcheck/t/002_cic.pl | 82 ++++++++++++++++++++++
src/test/perl/PostgresNode.pm | 126 ++++++++++++++++++++++++++++++++++
2 files changed, 208 insertions(+)
create mode 100644 contrib/amcheck/t/002_cic.pl
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
new file mode 100644
index 0000000000..31ce1a9a9c
--- /dev/null
+++ b/contrib/amcheck/t/002_cic.pl
@@ -0,0 +1,82 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 4;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+#
+# Stress CIC with pgbench
+#
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=200');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=200',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent transactions',
+ {
+ '002_pgbench_concurrent_transaction' =>
+ q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ ),
+ '002_pgbench_concurrent_transaction_savepoints' =>
+ q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ SAVEPOINT s1;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8158ea5b2f..631eb5618e 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1912,6 +1912,132 @@ sub background_psql
=pod
+=item $node->pgbench($opts, $stat, $out, $err, $name, $files, @args)
+
+Invoke pgbench, with parameters and files.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $stat
+
+Expected exit status.
+
+=item $out
+
+Reference to a regexp list that must match stdout.
+
+=item $err
+
+Reference to a regexp list that must match stderr.
+
+=item $name
+
+Name of test for error messages.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=item @args
+
+Further raw options or arguments.
+
+=back
+
+=cut
+sub pgbench
+{
+ my ($self, $opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $self->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+=pod
+
+=item $node->background_pgbench($dbname, \$stdin, \$stdout, $timer, $files, %opts) => harness
+
+Invoke B<pgbench> in background alike background_psql.
+
+=cut
+sub background_pgbench
+{
+ my ($self, $dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $self->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
+=pod
+
=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
--
2.24.3 (Apple Git-128)
=
v13-0006-Do-CIC-test-time-based-to-ensure-bug-repro.patch (application/octet-stream, 1.6K) download
From 25004bdee3cf7688805ac3f5d304e5a3ee48e932 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 15 Aug 2021 11:04:35 +0000
Subject: [PATCH v13 6/6] Do CIC test time-based to ensure bug repro
Not for commit.
---
contrib/amcheck/t/002_cic.pl | 4 ++--
contrib/amcheck/t/003_cic_2pc.pl | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
index 31ce1a9a9c..e4d1ad1461 100644
--- a/contrib/amcheck/t/002_cic.pl
+++ b/contrib/amcheck/t/002_cic.pl
@@ -41,11 +41,11 @@ my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_ou
SELECT bt_index_check('idx',true);
)
},
- '--no-vacuum --client=1 --transactions=200');
+ '--no-vacuum --client=1 --time=10');
# Run pgbench.
$node->pgbench(
- '--no-vacuum --client=5 --transactions=200',
+ '--no-vacuum --client=5 --time=10',
0,
[qr{actually processed}],
[qr{^$}],
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index f97ecd71d0..fa1a4994d7 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -114,11 +114,11 @@ my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_ou
SELECT bt_index_check('idx',true);
)
},
- '--no-vacuum --client=1 --transactions=100');
+ '--no-vacuum --client=1 --time=10');
# Run pgbench.
$node->pgbench(
- '--no-vacuum --client=5 --transactions=100',
+ '--no-vacuum --client=5 --time=10',
0,
[qr{actually processed}],
[qr{^$}],
--
2.24.3 (Apple Git-128)
=
v13-0005-Reorder-steps-of-2PC-preparation.patch (application/octet-stream, 1.7K) download
From a5b4a04ad079d834d77c82fd29c7a7a4d04288dc Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 7 Aug 2021 20:27:14 +0500
Subject: [PATCH v13 5/6] Reorder steps of 2PC preparation
Currently we remove xid from PGPROC being referenced by locks.
We can do so only when locks are transfered to 2PC PGPROC.
---
src/backend/access/transam/xact.c | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 387f80419a..bebd08f598 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2495,13 +2495,6 @@ PrepareTransaction(void)
/* Reset XactLastRecEnd until the next transaction writes something */
XactLastRecEnd = 0;
- /*
- * Let others know about no transaction in progress by me. This has to be
- * done *after* the prepared transaction has been marked valid, else
- * someone may think it is unlocked and recyclable.
- */
- ProcArrayClearTransaction(MyProc);
-
/*
* In normal commit-processing, this is all non-critical post-transaction
* cleanup. When the transaction is prepared, however, it's important
@@ -2535,6 +2528,15 @@ PrepareTransaction(void)
PostPrepare_MultiXact(xid);
PostPrepare_Locks(xid);
+
+ /*
+ * Let others know about no transaction in progress by me. This has to be
+ * done *after* the prepared transaction has been marked valid and locks
+ * transfered to new PGPROC, else someone may think it is unlocked and
+ * recyclable.
+ */
+ ProcArrayClearTransaction(MyProc);
+
PostPrepare_PredicateLocks(xid);
ResourceOwnerRelease(TopTransactionResourceOwner,
--
2.24.3 (Apple Git-128)
=
v13-0004-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxi.patch (application/octet-stream, 7.1K) download
From e0be4db4c31f5e26662493edd6d051e475004bf7 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v13 4/6] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2PC
Transaction having vxid can be converted to 2PC after GetLockConflicts()
call. To be sure we await it's completition we setup lxid\backend id
if 2PC PGPROCs so that we can find what to wait for if we only have vxid.
---
src/backend/access/transam/twophase.c | 46 ++++++++++++++-
src/backend/storage/lmgr/lock.c | 83 +++++++++++++++++++++------
src/include/access/twophase.h | 2 +
3 files changed, 110 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 2156de187c..a1ed4c6f5d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +847,45 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts.
+ * If more then one 2PC have same vxid - return any and set overflown to
+ * true.
+ */
+TransactionId
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid, bool *overflown)
+{
+ int i;
+ TransactionId result = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
+ VirtualTransactionId proc_vxid;
+ GET_VXID_FROM_PGPROC(proc_vxid, *proc);
+
+ if (VirtualTransactionIdEquals(vxid, proc_vxid))
+ {
+ if (result != InvalidTransactionId)
+ {
+ /* Already has candidate - need to set overflown */
+ *overflown = true;
+ return result;
+ }
+ /* Found one, but need to check that there is no others */
+ result = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..f55324d0da 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -387,6 +387,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool WaitPreparedXact(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -4529,6 +4530,56 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * WaitPreparedXact
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * two-phase procarray entries.
+ */
+static bool WaitPreparedXact(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ bool overflown = false;
+
+ /* There is no point to wait for 2PCs if you have no 2PCs. */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ do
+ {
+ /* Clear from restart attempts. */
+ if(overflown)
+ {
+ xid = InvalidTransactionId;
+ overflown = false;
+ }
+
+ /* If we have no xid - try to find one. */
+ if (!TransactionIdIsValid(xid))
+ xid = TwoPhaseGetXidByVXid(vxid, &overflown);
+
+ /* There is no 2PC with this vxid - exit */
+ if (!TransactionIdIsValid(xid))
+ {
+ Assert(!overflown);
+ return true;
+ }
+
+ /* Wait for xid completition */
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ return false;
+ LockRelease(&tag, ShareLock, false);
+
+ /* if there was multiple 2PCs with same vxid - start over. */
+ /* This never happens if we had a known xid as an argument. */
+ } while (overflown);
+
+ return true;
+}
+
/*
* VirtualXactLock
*
@@ -4541,25 +4592,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return WaitPreparedXact(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4573,7 +4617,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return WaitPreparedXact(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4587,9 +4631,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return WaitPreparedXact(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4641,7 +4688,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return WaitPreparedXact(vxid, xid, wait);
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index e27e1a8fe8..03d4f73868 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -60,4 +60,6 @@ extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
extern bool LookupGXact(const char *gid, XLogRecPtr prepare_at_lsn,
TimestampTz origin_prepare_timestamp);
+
+extern TransactionId TwoPhaseGetXidByVXid(VirtualTransactionId vxid, bool *overflown);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v13-0003-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 4.6K) download
From 69618d8aa7e15ffb3c57c481155c77a99c82807b Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 8 Aug 2021 14:46:50 +0500
Subject: [PATCH v13 3/6] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/t/003_cic_2pc.pl | 156 +++++++++++++++++++++++++++++++
1 file changed, 156 insertions(+)
create mode 100644 contrib/amcheck/t/003_cic_2pc.pl
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
new file mode 100644
index 0000000000..f97ecd71d0
--- /dev/null
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -0,0 +1,156 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY with prepared transactions
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(180);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint1
+);
+pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(180);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint2
+);
+pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint3
+);
+pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=100');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=100',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ ),
+ '002_pgbench_concurrent_2pc_savepoint' =>
+ q(
+ BEGIN;
+ SAVEPOINT s1;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v13-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcac.patch (application/octet-stream, 3.3K) download
From 599cedfb433aa22fa4ebd1f27267de98b378d157 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 14:40:16 +0500
Subject: [PATCH v13 2/6] PoC fix for race in RelationBuildDesc() and relcache
invalidation
Index indisready may be changed during RelationBuildDesc(), but invalidation
message for this can be consumed by some other call of RelationBuildDesc().
To prevent this we set up list of invalidated relation descriptions.
---
src/backend/utils/cache/relcache.c | 41 ++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 13d9994af3..d6165be5a3 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,21 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+
+/*
+ * InProgressRels is a stack of current RelationBuildDesc() calls
+ * that can consume incalidation messages and put borken desc into cache.
+ */
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1045,15 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ /* Register ourselves as in progress of RelationBuildDesc for invalidation messages */
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1234,13 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ /* We got invalidation message, need to rebuild */
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ /* inProgress is in fact the stack, we can safely remove it's top */
+ inProgress[in_progress_offset].relid = InvalidOid;
+ Assert(inProgress[in_progress_offset + 1].relid == InvalidOid);
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2805,6 +2833,14 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ /* Any RelationBuildDesc() on the stack must start over. */
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
@@ -2845,6 +2881,7 @@ RelationCacheInvalidate(void)
List *rebuildFirstList = NIL;
List *rebuildList = NIL;
ListCell *l;
+ int i;
/*
* Reload relation mapping data before starting to reconstruct cache.
@@ -2931,6 +2968,10 @@ RelationCacheInvalidate(void)
RelationClearRelation(relation, true);
}
list_free(rebuildList);
+
+ /* Any RelationBuildDesc() on the stack must start over. */
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ inProgress[i].invalidated = true;
}
/*
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-29 18:09 ` Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Andres Freund @ 2021-08-29 18:09 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
Hi,
On 2021-08-29 12:27:31 +0500, Andrey Borodin wrote:
> > 15 авг. 2021 г., в 18:45, Andres Freund <andres@anarazel.de> написал(а):
> > On 2021-08-15 16:09:37 +0500, Andrey Borodin wrote:
> >> @@ -4573,7 +4649,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
> >> */
> >> proc = BackendIdGetProc(vxid.backendId);
> >> if (proc == NULL)
> >> - return true;
> >> + return WaitXact(vxid, InvalidTransactionId, wait);
> >>
> >> /*
> >> * We must acquire this lock before checking the backendId and lxid
> >> @@ -4587,9 +4663,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
> >> || proc->fpLocalTransactionId != vxid.localTransactionId)
> >> {
> >> LWLockRelease(&proc->fpInfoLock);
> >> - return true;
> >> + return WaitXact(vxid, InvalidTransactionId, wait);
> >> }
> >
> > It seems like it's going to add a substantial amount of work even when
> > no 2PC xacts are involved?
> Only if 2PCs are enabled.
I don't think that's good enough. Plenty of systems have 2PC enabled but very
few if any transactions end up as 2PC ones.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
@ 2021-08-29 18:38 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-29 18:38 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
> 29 авг. 2021 г., в 23:09, Andres Freund <andres@anarazel.de> написал(а):
>>>
>>> It seems like it's going to add a substantial amount of work even when
>>> no 2PC xacts are involved?
>> Only if 2PCs are enabled.
>
> I don't think that's good enough. Plenty of systems have 2PC enabled but very
> few if any transactions end up as 2PC ones.
I could not compose significantly better solution. Let's consider what we actually pay for.
GetLockConflicts() returns xids for 2PC and vxids for any other transaction. Then we wait for each other one-by-one.
GetLockConflicts() cannot return xids for regular transactions: these xids may not have xid lock yet. So we have to work with vxid.
Each wait case can resort to set of three possible actions: test/wait for xid (cheap), test/wait for vxid (relatively cheap), search xid by vxid (expensive).
When we have to wait for known-2PC we wait on xid (cheap).
When we have vxid:
1. If backend is not on the same vxid, we have to search xid by vxid (expensive). Then wait on obtained xid if any.
2. Either way wait for vxid, then for xid (cheap * 2).
So the really slow part is searching xids for long-gone vxids that may not have a xid at all.
We simply have no good mechanics to say that this vxid did not have a xid, if vxid is long gone.
Best optimisation I can imagine here is to gather all vxids with unknown xids and search for them in one call to TwoPhaseGetXidByVXid() with one LWLockAcquire(TwoPhaseStateLock, LW_SHARED).
Does it worth the complexity?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-09-08 06:45 ` Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-09-08 06:45 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Sun, Aug 29, 2021 at 11:38:03PM +0500, Andrey Borodin wrote:
> > 29 авг. 2021 г., в 23:09, Andres Freund <andres@anarazel.de> написал(а):
> >>> It seems like it's going to add a substantial amount of work even when
> >>> no 2PC xacts are involved?
> >> Only if 2PCs are enabled.
> >
> > I don't think that's good enough. Plenty of systems have 2PC enabled but very
> > few if any transactions end up as 2PC ones.
> Best optimisation I can imagine here is to gather all vxids with unknown xids and search for them in one call to TwoPhaseGetXidByVXid() with one LWLockAcquire(TwoPhaseStateLock, LW_SHARED).
>
> Does it worth the complexity?
https://www.postgresql.org/search/?m=1&q=TwoPhaseStateLock&l=&d=-1&s=r
suggests this is the first postgresql.org discussion of TwoPhaseStateLock as a
bottleneck. Nonetheless, if Andres Freund finds it's worth the complexity,
then I'm content with it. I'd certainly expect some performance benefit.
Andres, what do you think?
We could start with an unlocked scan of the twophase shared memory. If the
unlocked scan finds a potential match, acquire TwoPhaseStateLock and repeat
the scan. Otherwise, we don't need the locked scan, because we can deduce
that the locked scan would find nothing. I'm not fond of relying on such
reasoning without a known-strong performance need, but it's an alternative.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-09-20 04:41 ` Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-09-20 04:41 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Mon, Aug 23, 2021 at 10:38:00PM +0500, Andrey Borodin wrote:
> --- a/src/backend/access/transam/twophase.c
> +++ b/src/backend/access/transam/twophase.c
> @@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
> proc->pgprocno = gxact->pgprocno;
> SHMQueueElemInit(&(proc->links));
> proc->waitStatus = PROC_WAIT_STATUS_OK;
> - /* We set up the gxact's VXID as InvalidBackendId/XID */
> - proc->lxid = (LocalTransactionId) xid;
> + /* We set up the gxact's VXID as real for CIC purposes */
> + proc->lxid = MyProc->lxid;
This breaks the case where the server restarted after PREPARE TRANSACTION.
MyProc->lxid is 0 in the startup process, and LocalTransactionIdIsValid(0) is
false. I'm attaching a test case addition. Can you repair this?
> proc->xid = xid;
> Assert(proc->xmin == InvalidTransactionId);
> proc->delayChkpt = false;
> proc->statusFlags = 0;
> proc->pid = 0;
> - proc->backendId = InvalidBackendId;
> + /* May be backendId of startup process */
> + proc->backendId = MyBackendId;
Incidentally, MyBackendId of startup process depends on other facts. When
using hot standby, InitRecoveryTransactionEnvironment() sets MyBackendId=1.
Otherwise, including clean startup of a non-standby node, MyBackendId is
InvalidBackendId. This may be harmless. I didn't know about it.
On Tue, Sep 07, 2021 at 11:45:15PM -0700, Noah Misch wrote:
> On Sun, Aug 29, 2021 at 11:38:03PM +0500, Andrey Borodin wrote:
> > > 29 авг. 2021 г., в 23:09, Andres Freund <andres@anarazel.de> написал(а):
> > >>> It seems like it's going to add a substantial amount of work even when
> > >>> no 2PC xacts are involved?
> > >> Only if 2PCs are enabled.
> > >
> > > I don't think that's good enough. Plenty of systems have 2PC enabled but very
> > > few if any transactions end up as 2PC ones.
>
> > Best optimisation I can imagine here is to gather all vxids with unknown xids and search for them in one call to TwoPhaseGetXidByVXid() with one LWLockAcquire(TwoPhaseStateLock, LW_SHARED).
> >
> > Does it worth the complexity?
>
> https://www.postgresql.org/search/?m=1&q=TwoPhaseStateLock&l=&d=-1&s=r
> suggests this is the first postgresql.org discussion of TwoPhaseStateLock as a
> bottleneck. Nonetheless, if Andres Freund finds it's worth the complexity,
> then I'm content with it. I'd certainly expect some performance benefit.
> Andres, what do you think?
A few more benefits (beyond lock contention) come to mind:
- Looking at the three VirtualXactLock() callers, waiting for final
disposition of prepared transactions is necessary for
WaitForLockersMultiple(), disadvantageous for WaitForOlderSnapshots(), and
dead code for ResolveRecoveryConflictWithVirtualXIDs(). In
WaitForOlderSnapshots(), PREPARE is as good as COMMIT/ABORT, because a
prepared transaction won't do further database reads. Waiting on the
prepared transaction there could give CIC an arbitrarily-long, needless
delay. ResolveRecoveryConflictWithVirtualXIDs() will never wait on a
prepared transaction, because prepared transactions hold no locks during
recovery. (If a prepared transaction originally acquired
AccessExclusiveLock, the startup process holds that lock on its behalf.)
Coordinating the XID search at a higher layer would let us change
WaitForLockersMultiple() without changing the others.
- v13 WaitPreparedXact() experiences starvation when a steady stream of
prepared transactions have the same VXID. Since VXID reuse entails
reconnecting, starvation will be unnoticeable in systems that follow best
practices around connection lifespan. The 2021-08-23 patch version didn't
have that hazard.
None of those benefits clearly justify the complexity, but they're relevant to
the decision.
Attachments:
test-2pc-wait-after-recovery-v0.patch (text/plain, 1.3K) download
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index fa1a499..9190dca 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -9,7 +9,7 @@ use Config;
use PostgresNode;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 6;
my ($node, $result);
@@ -95,6 +95,37 @@ is($result, '0', 'bt_index_check checks index');
#
+# Server restart shall not change whether prepared xact blocks CIC
+#
+
+$node->safe_psql('postgres', q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+PREPARE TRANSACTION 'spans_restart';
+BEGIN;
+CREATE TABLE unused ();
+PREPARE TRANSACTION 'persists_forever';
+));
+$node->restart;
+
+$reindex_in = '';
+$reindex_out = '';
+$reindex_timer = IPC::Run::timeout(180);
+$reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
+$reindex_h->finish;
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
# Test 2. Stress CIC+2PC with pgbench
#
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-09-25 17:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-09-25 17:25 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 20 сент. 2021 г., в 09:41, Noah Misch <noah@leadboat.com> написал(а):
>
> On Mon, Aug 23, 2021 at 10:38:00PM +0500, Andrey Borodin wrote:
>> --- a/src/backend/access/transam/twophase.c
>> +++ b/src/backend/access/transam/twophase.c
>> @@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
>> proc->pgprocno = gxact->pgprocno;
>> SHMQueueElemInit(&(proc->links));
>> proc->waitStatus = PROC_WAIT_STATUS_OK;
>> - /* We set up the gxact's VXID as InvalidBackendId/XID */
>> - proc->lxid = (LocalTransactionId) xid;
>> + /* We set up the gxact's VXID as real for CIC purposes */
>> + proc->lxid = MyProc->lxid;
>
> This breaks the case where the server restarted after PREPARE TRANSACTION.
> MyProc->lxid is 0 in the startup process, and LocalTransactionIdIsValid(0) is
> false. I'm attaching a test case addition. Can you repair this?
Yup. Indeed, that's a bug. Root cause it that GetLockConflicts() does not try to extract real xid from gxact's PGPROC, while vxid is not valid.
I see two ways to solve this:
1. Always set valid vxid, but fake 'vxid from xid' for gxact.
2. Teach GetLockConflicts() to use xid if vxid is invalid.
Both ways lead to identical GetLockConflicts() output.
PFA implementation of approach 1.
>> proc->xid = xid;
>> Assert(proc->xmin == InvalidTransactionId);
>> proc->delayChkpt = false;
>> proc->statusFlags = 0;
>> proc->pid = 0;
>> - proc->backendId = InvalidBackendId;
>> + /* May be backendId of startup process */
>> + proc->backendId = MyBackendId;
>
> Incidentally, MyBackendId of startup process depends on other facts. When
> using hot standby, InitRecoveryTransactionEnvironment() sets MyBackendId=1.
> Otherwise, including clean startup of a non-standby node, MyBackendId is
> InvalidBackendId. This may be harmless. I didn't know about it.
I think we should avoid using backendId during startup. The backend itself has nothing to do with the transaction.
> On Tue, Sep 07, 2021 at 11:45:15PM -0700, Noah Misch wrote:
>> On Sun, Aug 29, 2021 at 11:38:03PM +0500, Andrey Borodin wrote:
>>>> 29 авг. 2021 г., в 23:09, Andres Freund <andres@anarazel.de> написал(а):
>>>>>> It seems like it's going to add a substantial amount of work even when
>>>>>> no 2PC xacts are involved?
>>>>> Only if 2PCs are enabled.
>>>>
>>>> I don't think that's good enough. Plenty of systems have 2PC enabled but very
>>>> few if any transactions end up as 2PC ones.
>>
>>> Best optimisation I can imagine here is to gather all vxids with unknown xids and search for them in one call to TwoPhaseGetXidByVXid() with one LWLockAcquire(TwoPhaseStateLock, LW_SHARED).
>>>
>>> Does it worth the complexity?
>>
>> https://www.postgresql.org/search/?m=1&q=TwoPhaseStateLock&l=&d=-1&s=r
>> suggests this is the first postgresql.org discussion of TwoPhaseStateLock as a
>> bottleneck. Nonetheless, if Andres Freund finds it's worth the complexity,
>> then I'm content with it. I'd certainly expect some performance benefit.
>> Andres, what do you think?
>
> A few more benefits (beyond lock contention) come to mind:
>
> - Looking at the three VirtualXactLock() callers, waiting for final
> disposition of prepared transactions is necessary for
> WaitForLockersMultiple(), disadvantageous for WaitForOlderSnapshots(), and
> dead code for ResolveRecoveryConflictWithVirtualXIDs(). In
> WaitForOlderSnapshots(), PREPARE is as good as COMMIT/ABORT, because a
> prepared transaction won't do further database reads. Waiting on the
> prepared transaction there could give CIC an arbitrarily-long, needless
> delay. ResolveRecoveryConflictWithVirtualXIDs() will never wait on a
> prepared transaction, because prepared transactions hold no locks during
> recovery. (If a prepared transaction originally acquired
> AccessExclusiveLock, the startup process holds that lock on its behalf.)
> Coordinating the XID search at a higher layer would let us change
> WaitForLockersMultiple() without changing the others.
BTW WaitForOlderSnapshots() is used in ATExecDetachPartitionFinalize(). Probably, we could indicate to VirtualXactLock() if we want to wait on 2PC or not. Does it make sense?
>
>
> - v13 WaitPreparedXact() experiences starvation when a steady stream of
> prepared transactions have the same VXID. Since VXID reuse entails
> reconnecting, starvation will be unnoticeable in systems that follow best
> practices around connection lifespan. The 2021-08-23 patch version didn't
> have that hazard.
I think the probability of such a stream is abysmal. You not only need a stream of 2PC with the same vxid, but a stream of overlapping 2PC with the same vxid. And the most critical thing that can happen - CIC waiting for the stream to became one-2PC-at-a-time for a moment.
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (6 files, .tar.gz)
v14-0001-Introduce-TAP-test-for-CIC.patch (application/octet-stream, 5.6K) download
From c776952049b52b484cf11e2db24233882adc5493 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v14 1/6] Introduce TAP test for CIC
Implement some pgbench probablistic testing to stress out CREATE INDEX CONCURRENTLY.
---
contrib/amcheck/t/002_cic.pl | 82 ++++++++++++++++++++++
src/test/perl/PostgresNode.pm | 126 ++++++++++++++++++++++++++++++++++
2 files changed, 208 insertions(+)
create mode 100644 contrib/amcheck/t/002_cic.pl
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
new file mode 100644
index 0000000000..31ce1a9a9c
--- /dev/null
+++ b/contrib/amcheck/t/002_cic.pl
@@ -0,0 +1,82 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 4;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+#
+# Stress CIC with pgbench
+#
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=200');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=200',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent transactions',
+ {
+ '002_pgbench_concurrent_transaction' =>
+ q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ ),
+ '002_pgbench_concurrent_transaction_savepoints' =>
+ q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ SAVEPOINT s1;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8158ea5b2f..631eb5618e 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1912,6 +1912,132 @@ sub background_psql
=pod
+=item $node->pgbench($opts, $stat, $out, $err, $name, $files, @args)
+
+Invoke pgbench, with parameters and files.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $stat
+
+Expected exit status.
+
+=item $out
+
+Reference to a regexp list that must match stdout.
+
+=item $err
+
+Reference to a regexp list that must match stderr.
+
+=item $name
+
+Name of test for error messages.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=item @args
+
+Further raw options or arguments.
+
+=back
+
+=cut
+sub pgbench
+{
+ my ($self, $opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $self->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+=pod
+
+=item $node->background_pgbench($dbname, \$stdin, \$stdout, $timer, $files, %opts) => harness
+
+Invoke B<pgbench> in background alike background_psql.
+
+=cut
+sub background_pgbench
+{
+ my ($self, $dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $self->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
+=pod
+
=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
--
2.24.3 (Apple Git-128)
=
v14-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcac.patch (application/octet-stream, 3.3K) download
From 599cedfb433aa22fa4ebd1f27267de98b378d157 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 14:40:16 +0500
Subject: [PATCH v14 2/6] PoC fix for race in RelationBuildDesc() and relcache
invalidation
Index indisready may be changed during RelationBuildDesc(), but invalidation
message for this can be consumed by some other call of RelationBuildDesc().
To prevent this we set up list of invalidated relation descriptions.
---
src/backend/utils/cache/relcache.c | 41 ++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 13d9994af3..d6165be5a3 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,21 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+
+/*
+ * InProgressRels is a stack of current RelationBuildDesc() calls
+ * that can consume incalidation messages and put borken desc into cache.
+ */
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1045,15 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ /* Register ourselves as in progress of RelationBuildDesc for invalidation messages */
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1234,13 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ /* We got invalidation message, need to rebuild */
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ /* inProgress is in fact the stack, we can safely remove it's top */
+ inProgress[in_progress_offset].relid = InvalidOid;
+ Assert(inProgress[in_progress_offset + 1].relid == InvalidOid);
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2805,6 +2833,14 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ /* Any RelationBuildDesc() on the stack must start over. */
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
@@ -2845,6 +2881,7 @@ RelationCacheInvalidate(void)
List *rebuildFirstList = NIL;
List *rebuildList = NIL;
ListCell *l;
+ int i;
/*
* Reload relation mapping data before starting to reconstruct cache.
@@ -2931,6 +2968,10 @@ RelationCacheInvalidate(void)
RelationClearRelation(relation, true);
}
list_free(rebuildList);
+
+ /* Any RelationBuildDesc() on the stack must start over. */
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ inProgress[i].invalidated = true;
}
/*
--
2.24.3 (Apple Git-128)
=
v14-0003-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 5.5K) download
From 9ad741421cba4c6906078f76cb7d118cdde96eff Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 8 Aug 2021 14:46:50 +0500
Subject: [PATCH v14 3/6] Introduce TAP test for 2PC with CIC behavior
Currently there are two set of tests:
1. Deterministic reproduction of the bug with 3 psql sessions
This set also checks that 2PC interacts with CIC correctly
even after restart.
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/t/003_cic_2pc.pl | 187 +++++++++++++++++++++++++++++++
1 file changed, 187 insertions(+)
create mode 100644 contrib/amcheck/t/003_cic_2pc.pl
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
new file mode 100644
index 0000000000..101a0fea0b
--- /dev/null
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -0,0 +1,187 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY with prepared transactions
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 6;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(180);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint1
+);
+pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(180);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint2
+);
+pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint3
+);
+pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Server restart shall not change whether prepared xact blocks CIC
+#
+
+$node->safe_psql('postgres', q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+PREPARE TRANSACTION 'spans_restart';
+BEGIN;
+CREATE TABLE unused ();
+PREPARE TRANSACTION 'persists_forever';
+));
+$node->restart;
+
+$reindex_in = '';
+$reindex_out = '';
+$reindex_timer = IPC::Run::timeout(180);
+$reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
+$reindex_h->finish;
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=100');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=100',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ ),
+ '002_pgbench_concurrent_2pc_savepoint' =>
+ q(
+ BEGIN;
+ SAVEPOINT s1;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v14-0004-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxi.patch (application/octet-stream, 7.4K) download
From 5a6b6553dfd91d1d42e6fb42a43d172915a393e5 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v14 4/6] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2PC
Transaction having vxid can be converted to 2PC after GetLockConflicts()
call. To be sure we await it's completition we setup lxid\backend id
if 2PC PGPROCs so that we can find what to wait for if we only have vxid.
---
src/backend/access/transam/twophase.c | 58 ++++++++++++++++++-
src/backend/storage/lmgr/lock.c | 83 +++++++++++++++++++++------
src/include/access/twophase.h | 2 +
3 files changed, 122 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 2156de187c..1c5dd0342d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,27 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ /* We set up the gxact's VXID as real for CIC purposes */
+ if (LocalTransactionIdIsValid(MyProc->lxid))
+ {
+ proc->lxid = MyProc->lxid;
+ proc->backendId = MyBackendId;
+ }
+ else
+ {
+ /*
+ * Probably we are in startup process.
+ * Set any VXID that will be recognized as VirtualTransactionIdIsValid
+ * in GetLockConflicts().
+ */
+ proc->lxid = xid;
+ proc->backendId = InvalidBackendId;
+ }
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +859,45 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts.
+ * If more then one 2PC have same vxid - return any and set overflown to
+ * true.
+ */
+TransactionId
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid, bool *overflown)
+{
+ int i;
+ TransactionId result = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
+ VirtualTransactionId proc_vxid;
+ GET_VXID_FROM_PGPROC(proc_vxid, *proc);
+
+ if (VirtualTransactionIdEquals(vxid, proc_vxid))
+ {
+ if (result != InvalidTransactionId)
+ {
+ /* Already has candidate - need to set overflown */
+ *overflown = true;
+ return result;
+ }
+ /* Found one, but need to check that there is no others */
+ result = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..f55324d0da 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -387,6 +387,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool WaitPreparedXact(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -4529,6 +4530,56 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * WaitPreparedXact
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * two-phase procarray entries.
+ */
+static bool WaitPreparedXact(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ bool overflown = false;
+
+ /* There is no point to wait for 2PCs if you have no 2PCs. */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ do
+ {
+ /* Clear from restart attempts. */
+ if(overflown)
+ {
+ xid = InvalidTransactionId;
+ overflown = false;
+ }
+
+ /* If we have no xid - try to find one. */
+ if (!TransactionIdIsValid(xid))
+ xid = TwoPhaseGetXidByVXid(vxid, &overflown);
+
+ /* There is no 2PC with this vxid - exit */
+ if (!TransactionIdIsValid(xid))
+ {
+ Assert(!overflown);
+ return true;
+ }
+
+ /* Wait for xid completition */
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ return false;
+ LockRelease(&tag, ShareLock, false);
+
+ /* if there was multiple 2PCs with same vxid - start over. */
+ /* This never happens if we had a known xid as an argument. */
+ } while (overflown);
+
+ return true;
+}
+
/*
* VirtualXactLock
*
@@ -4541,25 +4592,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return WaitPreparedXact(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4573,7 +4617,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return WaitPreparedXact(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4587,9 +4631,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return WaitPreparedXact(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4641,7 +4688,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return WaitPreparedXact(vxid, xid, wait);
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index e27e1a8fe8..03d4f73868 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -60,4 +60,6 @@ extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
extern bool LookupGXact(const char *gid, XLogRecPtr prepare_at_lsn,
TimestampTz origin_prepare_timestamp);
+
+extern TransactionId TwoPhaseGetXidByVXid(VirtualTransactionId vxid, bool *overflown);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v14-0005-Reorder-steps-of-2PC-preparation.patch (application/octet-stream, 1.7K) download
From 88f4e21a9d8cec4fbe60029599d6001113d52585 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 7 Aug 2021 20:27:14 +0500
Subject: [PATCH v14 5/6] Reorder steps of 2PC preparation
Currently we remove xid from PGPROC being referenced by locks.
We can do so only when locks are transfered to 2PC PGPROC.
---
src/backend/access/transam/xact.c | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 387f80419a..bebd08f598 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2495,13 +2495,6 @@ PrepareTransaction(void)
/* Reset XactLastRecEnd until the next transaction writes something */
XactLastRecEnd = 0;
- /*
- * Let others know about no transaction in progress by me. This has to be
- * done *after* the prepared transaction has been marked valid, else
- * someone may think it is unlocked and recyclable.
- */
- ProcArrayClearTransaction(MyProc);
-
/*
* In normal commit-processing, this is all non-critical post-transaction
* cleanup. When the transaction is prepared, however, it's important
@@ -2535,6 +2528,15 @@ PrepareTransaction(void)
PostPrepare_MultiXact(xid);
PostPrepare_Locks(xid);
+
+ /*
+ * Let others know about no transaction in progress by me. This has to be
+ * done *after* the prepared transaction has been marked valid and locks
+ * transfered to new PGPROC, else someone may think it is unlocked and
+ * recyclable.
+ */
+ ProcArrayClearTransaction(MyProc);
+
PostPrepare_PredicateLocks(xid);
ResourceOwnerRelease(TopTransactionResourceOwner,
--
2.24.3 (Apple Git-128)
=
v14-0006-Do-CIC-test-time-based-to-ensure-bug-repro.patch (application/octet-stream, 1.6K) download
From 9b0258d76d71d73dc2abf05d844982025594894a Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 15 Aug 2021 11:04:35 +0000
Subject: [PATCH v14 6/6] Do CIC test time-based to ensure bug repro
Not for commit.
---
contrib/amcheck/t/002_cic.pl | 4 ++--
contrib/amcheck/t/003_cic_2pc.pl | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
index 31ce1a9a9c..e4d1ad1461 100644
--- a/contrib/amcheck/t/002_cic.pl
+++ b/contrib/amcheck/t/002_cic.pl
@@ -41,11 +41,11 @@ my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_ou
SELECT bt_index_check('idx',true);
)
},
- '--no-vacuum --client=1 --transactions=200');
+ '--no-vacuum --client=1 --time=10');
# Run pgbench.
$node->pgbench(
- '--no-vacuum --client=5 --transactions=200',
+ '--no-vacuum --client=5 --time=10',
0,
[qr{actually processed}],
[qr{^$}],
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index 101a0fea0b..9190dca06a 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -145,11 +145,11 @@ my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_ou
SELECT bt_index_check('idx',true);
)
},
- '--no-vacuum --client=1 --transactions=100');
+ '--no-vacuum --client=1 --time=10');
# Run pgbench.
$node->pgbench(
- '--no-vacuum --client=5 --transactions=100',
+ '--no-vacuum --client=5 --time=10',
0,
[qr{actually processed}],
[qr{^$}],
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-09-25 20:10 ` Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-09-25 20:10 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Sep 25, 2021 at 10:25:05PM +0500, Andrey Borodin wrote:
> > 20 сент. 2021 г., в 09:41, Noah Misch <noah@leadboat.com> написал(а):
> > On Mon, Aug 23, 2021 at 10:38:00PM +0500, Andrey Borodin wrote:
> >> --- a/src/backend/access/transam/twophase.c
> >> +++ b/src/backend/access/transam/twophase.c
> >> @@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
> >> proc->pgprocno = gxact->pgprocno;
> >> SHMQueueElemInit(&(proc->links));
> >> proc->waitStatus = PROC_WAIT_STATUS_OK;
> >> - /* We set up the gxact's VXID as InvalidBackendId/XID */
> >> - proc->lxid = (LocalTransactionId) xid;
> >> + /* We set up the gxact's VXID as real for CIC purposes */
> >> + proc->lxid = MyProc->lxid;
> >
> > This breaks the case where the server restarted after PREPARE TRANSACTION.
> > MyProc->lxid is 0 in the startup process, and LocalTransactionIdIsValid(0) is
> > false. I'm attaching a test case addition. Can you repair this?
> Yup. Indeed, that's a bug. Root cause it that GetLockConflicts() does not try to extract real xid from gxact's PGPROC, while vxid is not valid.
> I see two ways to solve this:
> 1. Always set valid vxid, but fake 'vxid from xid' for gxact.
> 2. Teach GetLockConflicts() to use xid if vxid is invalid.
> Both ways lead to identical GetLockConflicts() output.
> PFA implementation of approach 1.
That's reasonable. I'll queue a task to review the rest of the patch.
> > On Tue, Sep 07, 2021 at 11:45:15PM -0700, Noah Misch wrote:
> >> On Sun, Aug 29, 2021 at 11:38:03PM +0500, Andrey Borodin wrote:
> >>>> 29 авг. 2021 г., в 23:09, Andres Freund <andres@anarazel.de> написал(а):
> >>>>>> It seems like it's going to add a substantial amount of work even when
> >>>>>> no 2PC xacts are involved?
> >>>>> Only if 2PCs are enabled.
> >>>>
> >>>> I don't think that's good enough. Plenty of systems have 2PC enabled but very
> >>>> few if any transactions end up as 2PC ones.
> >>
> >>> Best optimisation I can imagine here is to gather all vxids with unknown xids and search for them in one call to TwoPhaseGetXidByVXid() with one LWLockAcquire(TwoPhaseStateLock, LW_SHARED).
> >>>
> >>> Does it worth the complexity?
> >>
> >> https://www.postgresql.org/search/?m=1&q=TwoPhaseStateLock&l=&d=-1&s=r
> >> suggests this is the first postgresql.org discussion of TwoPhaseStateLock as a
> >> bottleneck. Nonetheless, if Andres Freund finds it's worth the complexity,
> >> then I'm content with it. I'd certainly expect some performance benefit.
> >> Andres, what do you think?
> >
> > A few more benefits (beyond lock contention) come to mind:
> >
> > - Looking at the three VirtualXactLock() callers, waiting for final
> > disposition of prepared transactions is necessary for
> > WaitForLockersMultiple(), disadvantageous for WaitForOlderSnapshots(), and
> > dead code for ResolveRecoveryConflictWithVirtualXIDs(). In
> > WaitForOlderSnapshots(), PREPARE is as good as COMMIT/ABORT, because a
> > prepared transaction won't do further database reads. Waiting on the
> > prepared transaction there could give CIC an arbitrarily-long, needless
> > delay. ResolveRecoveryConflictWithVirtualXIDs() will never wait on a
> > prepared transaction, because prepared transactions hold no locks during
> > recovery. (If a prepared transaction originally acquired
> > AccessExclusiveLock, the startup process holds that lock on its behalf.)
> > Coordinating the XID search at a higher layer would let us change
> > WaitForLockersMultiple() without changing the others.
> BTW WaitForOlderSnapshots() is used in ATExecDetachPartitionFinalize(). Probably, we could indicate to VirtualXactLock() if we want to wait on 2PC or not. Does it make sense?
For both CIC and ATExecDetachPartitionFinalize(), one needs unlucky timing for
the operation to needlessly wait on a prepared xact. Specifically, the
needless wait arises if a PREPARE happens while WaitForOlderSnapshots() is
running, after its GetCurrentVirtualXIDs() and before its VirtualXactLock().
I would not choose to "indicate to VirtualXactLock() if we want to wait on 2PC
or not", but code like that wouldn't be too bad. I probably wouldn't remove
code like that if you chosen to write it.
The alternative I have in mind would work like the following pseudocode. I'm
leaning toward thinking it's not worth doing, since none of the three benefits
are known to be important. But maybe it is worth doing.
struct LockConflictData
{
/* VXIDs seen to have XIDs */
List *vxid_of_xid;
/* VXIDs without known XIDs */
List *vxid_no_xid;
/*
* XIDs. Has one entry per entry in vxid_of_xid, and it may have up to
* max_prepared_transactions additional entries.
*/
List *xid;
};
void
WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
{
struct LockConflictData holders;
List *latest_xid = NIL;
List *need_mapping = NIL;
ListCell *lc;
int total = 0;
int done = 0;
/* Collect the transactions we need to wait on */
foreach(lc, locktags)
{
LOCKTAG *locktag = lfirst(lc);
int count;
GetLockConflicts(&holders, locktag, lockmode,
progress ? &count : NULL);
}
/* wait on VXIDs known to have XIDs, and wait on known XIDs */
foreach(lc, holders.vxid_of_xid)
VirtualXactLock(lfirst_int(lc), true, NULL);
foreach(lc, holders.xid)
XactLockTableWait(lfirst_int(lc), other_params);
/* wait on remaining VXIDs, possibly discovering more XIDs */
foreach(lc, holders.vxid_no_xid)
{
VirtualTransactionId *v = lfirst(lc);
TransactionId xid = InvalidTransactionId;
/*
* Under this design, VirtualXactLock() would know nothing about 2PC,
* but it would gain the ability to return proc->xid of the waited
* proc. Under this design, VirtualTransactionId is always a local
* transaction, like it was before commit 8a54e12.
*/
VirtualXactLock(*v, true, &xid);
if (TransactionIdIsValid(xid))
latest_xid = lappend_int(late_xid, xid);
else
need_mapping = lappend_int(need_mapping, v);
}
/* wait on XIDs just discovered */
foreach(lc, latest_xid)
XactLockTableWait(lfirst_int(lc), other_params);
/*
* If we never saw an XID associated with a particular VXID, check whether
* the VXID became a prepared xact.
*/
latest_xid = TwoPhaseGetXidByVXid(need_mapping);
foreach(lc, latest_xid)
XactLockTableWait(lfirst_int(lc), other_params);
}
> > - v13 WaitPreparedXact() experiences starvation when a steady stream of
> > prepared transactions have the same VXID. Since VXID reuse entails
> > reconnecting, starvation will be unnoticeable in systems that follow best
> > practices around connection lifespan. The 2021-08-23 patch version didn't
> > have that hazard.
> I think the probability of such a stream is abysmal. You not only need a stream of 2PC with the same vxid, but a stream of overlapping 2PC with the same vxid. And the most critical thing that can happen - CIC waiting for the stream to became one-2PC-at-a-time for a moment.
You're probably right about that.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-17 15:12 ` Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-10-17 15:12 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Sep 25, 2021 at 01:10:12PM -0700, Noah Misch wrote:
> On Sat, Sep 25, 2021 at 10:25:05PM +0500, Andrey Borodin wrote:
> > > 20 сент. 2021 г., в 09:41, Noah Misch <noah@leadboat.com> написал(а):
> I'll queue a task to review the rest of the patch.
I think the attached version is ready for commit. Notable differences
vs. v14:
- Made TwoPhaseGetXidByVXid() stop leaking TwoPhaseStateLock when it found a
second match.
- Instead of moving the ProcArrayClearTransaction() call within
PrepareTransaction(), move the PostPrepare_Locks() call. I didn't find any
bug in the other way, but it's a good principle to maximize similarity with
CommitTransaction(). PostPrepare_Locks() has no counterpart in
CommitTransaction(), so that principle is indifferent to moving it.
- inval-build-race-v0.1.patch was incompatible with debug_discard_caches. The
being-built relation would always get invalidated, making
RelationBuildDesc() an infinite loop. I've fixed this by making
RelationCacheInvalidate() invalidate in-progress rels only when called for
sinval overflow, not when called for debug_discard_caches. This adds some
function arguments that only matter in assert builds. That's not great, but
InvalidateSystemCaches() is expensive anyway. I considered instead adding
functions HoldDebugInval() and ResumeDebugInval(), which RelationBuildDesc()
would use to suppress debug_discard_caches during any iteration after the
first. That didn't seem better.
- Discard the in-progress array after an ERROR during RelationBuildDesc().
- Made the relcache.c changes repalloc the list of in-progress rels as needed.
- Changed the background_pgbench args from ($dbname, $stdin, \$stdout, $timer,
$files, $opts) to ($opts, $files, \$stdout, $timer). $dbname was unused.
pgbench doesn't make interesting use of its stdin, so I dropped that arg
until we have a use case. $opts and $files seem akin to the $dbname arg of
background_psql, so I moved them to the front. I'm not sure that last
change was an improvement.
- Made 001_pgbench_with_server.pl use PostgresNode::pgbench(), rather than
duplicate code. Factored out a subroutine of PostgresNode::pgbench() and
PostgresNode::background_pgbench().
- Lots of comment and naming changes.
One thing not done here is to change the tests to use CREATE INDEX
CONCURRENTLY instead of REINDEX CONCURRENTLY, so they're back-patchable to v11
and earlier. I may do that before pushing, or I may just omit the tests from
older branches.
> > > - v13 WaitPreparedXact() experiences starvation when a steady stream of
> > > prepared transactions have the same VXID. Since VXID reuse entails
> > > reconnecting, starvation will be unnoticeable in systems that follow best
> > > practices around connection lifespan. The 2021-08-23 patch version didn't
> > > have that hazard.
> > I think the probability of such a stream is abysmal. You not only need a stream of 2PC with the same vxid, but a stream of overlapping 2PC with the same vxid. And the most critical thing that can happen - CIC waiting for the stream to became one-2PC-at-a-time for a moment.
>
> You're probably right about that.
I didn't know of the "stateP->nextLXID = nextLocalTransactionId;" in
CleanupInvalidationState(), which indeed makes this all but impossible.
On Sat, Aug 07, 2021 at 03:19:57PM -0700, Noah Misch wrote:
> On Sun, Aug 08, 2021 at 12:00:55AM +0500, Andrey Borodin wrote:
> > Changes:
> > 1. Added assert in step 2 (fix for missed invalidation message). I wonder how deep possibly could be RelationBuildDesc() inside RelationBuildDesc() inside RelationBuildDesc() ... ? If the depth is unlimited we, probably, need a better data structure.
>
> I don't know either, hence that quick data structure to delay the question.
> debug_discard_caches=3 may help answer the question. RelationBuildDesc()
> reads pg_constraint, which is !rd_isnailed. Hence, I expect one can at least
> get RelationBuildDesc("pg_constraint") inside RelationBuildDesc("user_table").
debug_discard_caches=5 yields a depth of eight when opening a relation having
a CHECK constraint:
my_rel_having_check_constraint
pg_constraint_conrelid_contypid_conname_index
pg_index
pg_constraint
pg_constraint
pg_constraint
pg_constraint
pg_constraint
While debug_discard_caches doesn't permit higher values, I think one could
reach depths greater than eight by, for example, having a number of sessions
invalidate pg_constraint as often as possible. Hence, I'm glad the code no
longer relies on a depth limit.
Attachments: download all (2 files, .tar.gz)
inval-build-race-v1.patch (text/plain, 25.4K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
new file mode 100644
index 0000000..dba301e
--- /dev/null
+++ b/contrib/amcheck/t/002_cic.pl
@@ -0,0 +1,80 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test CREATE INDEX CONCURRENTLY with concurrent modifications
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 4;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'lock_timeout = 180000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+#
+# Stress CIC with pgbench
+#
+
+# Run background pgbench with CIC. We cannot mix-in this script into single
+# pgbench: CIC will deadlock with itself occasionally.
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench(
+ '--no-vacuum --client=1 --transactions=200',
+ {
+ '002_pgbench_concurrent_cic' => q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ \$pgbench_out,
+ $pgbench_timer);
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=200',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent INSERTs',
+ {
+ '002_pgbench_concurrent_transaction' => q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ ),
+ '002_pgbench_concurrent_transaction_savepoints' => q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ SAVEPOINT s1;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result =
+ ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->stop;
+done_testing();
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 9352c68..dd8586a 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -628,7 +628,7 @@ LocalExecuteInvalidationMessage(SharedInvalidationMessage *msg)
int i;
if (msg->rc.relId == InvalidOid)
- RelationCacheInvalidate();
+ RelationCacheInvalidate(false);
else
RelationCacheInvalidateEntry(msg->rc.relId);
@@ -686,11 +686,17 @@ LocalExecuteInvalidationMessage(SharedInvalidationMessage *msg)
void
InvalidateSystemCaches(void)
{
+ InvalidateSystemCachesExtended(false);
+}
+
+void
+InvalidateSystemCachesExtended(bool debug_discard)
+{
int i;
InvalidateCatalogSnapshot();
ResetCatalogCaches();
- RelationCacheInvalidate(); /* gets smgr and relmap too */
+ RelationCacheInvalidate(debug_discard); /* gets smgr and relmap too */
for (i = 0; i < syscache_callback_count; i++)
{
@@ -759,7 +765,7 @@ AcceptInvalidationMessages(void)
if (recursion_depth < debug_discard_caches)
{
recursion_depth++;
- InvalidateSystemCaches();
+ InvalidateSystemCachesExtended(true);
recursion_depth--;
}
}
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 13d9994..b54c911 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -151,6 +151,24 @@ bool criticalSharedRelcachesBuilt = false;
static long relcacheInvalsReceived = 0L;
/*
+ * in_progress_list is a stack of ongoing RelationBuildDesc() calls. CREATE
+ * INDEX CONCURRENTLY makes catalog changes under ShareUpdateExclusiveLock.
+ * It critically relies on each backend absorbing those changes no later than
+ * next transaction start. Hence, RelationBuildDesc() loops until it finishes
+ * without accepting a relevant invalidation. (Most invalidation consumers
+ * don't do this.)
+ */
+typedef struct inprogressent
+{
+ Oid reloid; /* OID of relation being built */
+ bool invalidated; /* whether an invalidation arrived for it */
+} InProgressEnt;
+
+static InProgressEnt *in_progress_list;
+static int in_progress_list_len;
+static int in_progress_list_maxlen;
+
+/*
* eoxact_list[] stores the OIDs of relations that (might) need AtEOXact
* cleanup work. This list intentionally has limited size; if it overflows,
* we fall back to scanning the whole hashtable. There is no value in a very
@@ -1000,6 +1018,7 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1052,21 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ /* Register to catch invalidation messages */
+ if (in_progress_list_len >= in_progress_list_maxlen)
+ {
+ int allocsize;
+
+ allocsize = in_progress_list_maxlen * 2;
+ in_progress_list = repalloc(in_progress_list,
+ allocsize * sizeof(*in_progress_list));
+ in_progress_list_maxlen = allocsize;
+ }
+ in_progress_offset = in_progress_list_len++;
+ in_progress_list[in_progress_offset].reloid = targetRelId;
+retry:
+ in_progress_list[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1051,6 +1085,8 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
MemoryContextDelete(tmpcxt);
}
#endif
+ Assert(in_progress_offset + 1 == in_progress_list_len);
+ in_progress_list_len--;
return NULL;
}
@@ -1214,6 +1250,21 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
heap_freetuple(pg_class_tuple);
/*
+ * If an invalidation arrived mid-build, start over. Between here and the
+ * end of this function, don't add code that does or reasonably could read
+ * system catalogs. That range must be free from invalidation processing
+ * for the !insertIt case. For the insertIt case, RelationCacheInsert()
+ * will enroll this relation in ordinary relcache invalidation processing,
+ */
+ if (in_progress_list[in_progress_offset].invalidated)
+ {
+ RelationDestroyRelation(relation, false);
+ goto retry;
+ }
+ Assert(in_progress_offset + 1 == in_progress_list_len);
+ in_progress_list_len--;
+
+ /*
* Insert newly created relation into relcache hash table, if requested.
*
* There is one scenario in which we might find a hashtable entry already
@@ -2566,6 +2617,14 @@ RelationClearRelation(Relation relation, bool rebuild)
/* Build temporary entry, but don't link it into hashtable */
newrel = RelationBuildDesc(save_relid, false);
+
+ /*
+ * Between here and the end of the swap, don't add code that does or
+ * reasonably could read system catalogs. That range must be free
+ * from invalidation processing. See RelationBuildDesc() manipulation
+ * of in_progress_list.
+ */
+
if (newrel == NULL)
{
/*
@@ -2805,6 +2864,14 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+
+ for (i = 0; i < in_progress_list_len; i++)
+ if (in_progress_list[i].reloid == relationId)
+ in_progress_list[i].invalidated = true;
+ }
}
/*
@@ -2813,11 +2880,11 @@ RelationCacheInvalidateEntry(Oid relationId)
* and rebuild those with positive reference counts. Also reset the smgr
* relation cache and re-read relation mapping data.
*
- * This is currently used only to recover from SI message buffer overflow,
- * so we do not touch relations having new-in-transaction relfilenodes; they
- * cannot be targets of cross-backend SI updates (and our own updates now go
- * through a separate linked list that isn't limited by the SI message
- * buffer size).
+ * Apart from debug_discard_caches, this is currently used only to recover
+ * from SI message buffer overflow, so we do not touch relations having
+ * new-in-transaction relfilenodes; they cannot be targets of cross-backend
+ * SI updates (and our own updates now go through a separate linked list
+ * that isn't limited by the SI message buffer size).
*
* We do this in two phases: the first pass deletes deletable items, and
* the second one rebuilds the rebuildable items. This is essential for
@@ -2835,9 +2902,14 @@ RelationCacheInvalidateEntry(Oid relationId)
* second pass processes nailed-in-cache items before other nondeletable
* items. This should ensure that system catalogs are up to date before
* we attempt to use them to reload information about other open relations.
+ *
+ * After those two phases of work having immediate effects, we normally
+ * signal any RelationBuildDesc() on the stack to start over. However, we
+ * don't do this if called as part of debug_discard_caches. Otherwise,
+ * RelationBuildDesc() would become an infinite loop.
*/
void
-RelationCacheInvalidate(void)
+RelationCacheInvalidate(bool debug_discard)
{
HASH_SEQ_STATUS status;
RelIdCacheEnt *idhentry;
@@ -2845,6 +2917,7 @@ RelationCacheInvalidate(void)
List *rebuildFirstList = NIL;
List *rebuildList = NIL;
ListCell *l;
+ int i;
/*
* Reload relation mapping data before starting to reconstruct cache.
@@ -2931,6 +3004,11 @@ RelationCacheInvalidate(void)
RelationClearRelation(relation, true);
}
list_free(rebuildList);
+
+ if (!debug_discard)
+ /* Any RelationBuildDesc() on the stack must start over. */
+ for (i = 0; i < in_progress_list_len; i++)
+ in_progress_list[i].invalidated = true;
}
/*
@@ -3082,6 +3160,13 @@ AtEOXact_RelationCache(bool isCommit)
int i;
/*
+ * Forget in_progress_list. This is relevant when we're aborting due to
+ * an error during RelationBuildDesc().
+ */
+ Assert(in_progress_list_len == 0 || !isCommit);
+ in_progress_list_len = 0;
+
+ /*
* Unless the eoxact_list[] overflowed, we only need to examine the rels
* listed in it. Otherwise fall back on a hash_seq_search scan.
*
@@ -3228,6 +3313,14 @@ AtEOSubXact_RelationCache(bool isCommit, SubTransactionId mySubid,
int i;
/*
+ * Forget in_progress_list. This is relevant when we're aborting due to
+ * an error during RelationBuildDesc(). We don't commit subtransactions
+ * during RelationBuildDesc().
+ */
+ Assert(in_progress_list_len == 0 || !isCommit);
+ in_progress_list_len = 0;
+
+ /*
* Unless the eoxact_list[] overflowed, we only need to examine the rels
* listed in it. Otherwise fall back on a hash_seq_search scan. Same
* logic as in AtEOXact_RelationCache.
@@ -3775,6 +3868,7 @@ void
RelationCacheInitialize(void)
{
HASHCTL ctl;
+ int allocsize;
/*
* make sure cache memory context exists
@@ -3791,6 +3885,15 @@ RelationCacheInitialize(void)
&ctl, HASH_ELEM | HASH_BLOBS);
/*
+ * reserve enough in_progress_list slots for many cases
+ */
+ allocsize = 4;
+ in_progress_list =
+ MemoryContextAlloc(CacheMemoryContext,
+ allocsize * sizeof(*in_progress_list));
+ in_progress_list_maxlen = allocsize;
+
+ /*
* relation mapper needs to be initialized too
*/
RelationMapInitialize();
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index ef53f6b..7ca96e5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -14,54 +14,6 @@ my $node = PostgresNode->new('main');
$node->init;
$node->start;
-# invoke pgbench, with parameters:
-# $opts: options as a string to be split on spaces
-# $stat: expected exit status
-# $out: reference to a regexp list that must match stdout
-# $err: reference to a regexp list that must match stderr
-# $name: name of test for error messages
-# $files: reference to filename/contents dictionary
-# @args: further raw options or arguments
-sub pgbench
-{
- local $Test::Builder::Level = $Test::Builder::Level + 1;
-
- my ($opts, $stat, $out, $err, $name, $files, @args) = @_;
- my @cmd = ('pgbench', split /\s+/, $opts);
- my @filenames = ();
- if (defined $files)
- {
-
- # note: files are ordered for determinism
- for my $fn (sort keys %$files)
- {
- my $filename = $node->basedir . '/' . $fn;
- push @cmd, '-f', $filename;
-
- # cleanup file weight
- $filename =~ s/\@\d+$//;
-
- #push @filenames, $filename;
- # filenames are expected to be unique on a test
- if (-e $filename)
- {
- ok(0, "$filename must not already exist");
- unlink $filename or die "cannot unlink $filename: $!";
- }
- append_to_file($filename, $$files{$fn});
- }
- }
-
- push @cmd, @args;
-
- $node->command_checks_all(\@cmd, $stat, $out, $err, $name);
-
- # cleanup?
- #unlink @filenames or die "cannot unlink files (@filenames): $!";
-
- return;
-}
-
# tablespace for testing, because partitioned tables cannot use pg_default
# explicitly and we want to test that table creation with tablespace works
# for partitioned tables.
@@ -77,7 +29,7 @@ $node->safe_psql('postgres',
# Test concurrent OID generation via pg_enum_oid_index. This indirectly
# exercises LWLock and spinlock concurrency.
my $labels = join ',', map { "'l$_'" } 1 .. 1000;
-pgbench(
+$node->pgbench(
'--no-vacuum --client=5 --protocol=prepared --transactions=25',
0,
[qr{processed: 125/125}],
@@ -89,7 +41,7 @@ pgbench(
});
# Trigger various connection errors
-pgbench(
+$node->pgbench(
'no-such-database',
1,
[qr{^$}],
@@ -99,13 +51,13 @@ pgbench(
],
'no such database');
-pgbench(
+$node->pgbench(
'-S -t 1', 1, [],
[qr{Perhaps you need to do initialization}],
'run without init');
# Initialize pgbench tables scale 1
-pgbench(
+$node->pgbench(
'-i', 0,
[qr{^$}],
[
@@ -117,7 +69,7 @@ pgbench(
'pgbench scale 1 initialization',);
# Again, with all possible options
-pgbench(
+$node->pgbench(
'--initialize --init-steps=dtpvg --scale=1 --unlogged-tables --fillfactor=98 --foreign-keys --quiet --tablespace=regress_pgbench_tap_1_ts --index-tablespace=regress_pgbench_tap_1_ts --partitions=2 --partition-method=hash',
0,
[qr{^$}i],
@@ -134,7 +86,7 @@ pgbench(
'pgbench scale 1 initialization');
# Test interaction of --init-steps with legacy step-selection options
-pgbench(
+$node->pgbench(
'--initialize --init-steps=dtpvGvv --no-vacuum --foreign-keys --unlogged-tables --partitions=3',
0,
[qr{^$}],
@@ -151,7 +103,7 @@ pgbench(
'pgbench --init-steps');
# Run all builtin scripts, for a few transactions each
-pgbench(
+$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
. ' --connect -n -v -n',
0,
@@ -164,7 +116,7 @@ pgbench(
[qr{^$}],
'pgbench tpcb-like');
-pgbench(
+$node->pgbench(
'--transactions=20 --client=5 -M extended --builtin=si -C --no-vacuum -s 1',
0,
[
@@ -177,7 +129,7 @@ pgbench(
[qr{scale option ignored}],
'pgbench simple update');
-pgbench(
+$node->pgbench(
'-t 100 -c 7 -M prepared -b se --debug',
0,
[
@@ -203,7 +155,7 @@ my $nthreads = 2;
}
# run custom scripts
-pgbench(
+$node->pgbench(
"-t 100 -c 1 -j $nthreads -M prepared -n",
0,
[
@@ -233,7 +185,7 @@ COMMIT;
}
});
-pgbench(
+$node->pgbench(
'-n -t 10 -c 1 -M simple',
0,
[
@@ -254,7 +206,7 @@ COMMIT;
}
});
-pgbench(
+$node->pgbench(
'-n -t 10 -c 2 -M extended',
0,
[
@@ -285,7 +237,7 @@ $node->append_conf('postgresql.conf',
. "log_parameter_max_length = 0\n"
. "log_parameter_max_length_on_error = 0");
$node->reload;
-pgbench(
+$node->pgbench(
'-n -t1 -c1 -M prepared',
2,
[],
@@ -312,7 +264,7 @@ $node->append_conf('postgresql.conf',
"log_parameter_max_length = -1\n"
. "log_parameter_max_length_on_error = 64");
$node->reload;
-pgbench(
+$node->pgbench(
'-n -t1 -c1 -M prepared',
2,
[],
@@ -326,7 +278,7 @@ pgbench(
SELECT 1 / (random() / 2)::int, :one::int, :two::int;
}
});
-pgbench(
+$node->pgbench(
'-n -t1 -c1 -M prepared',
2,
[],
@@ -354,7 +306,7 @@ $node->append_conf('postgresql.conf',
. "log_parameter_max_length = 7\n"
. "log_parameter_max_length_on_error = -1");
$node->reload;
-pgbench(
+$node->pgbench(
'-n -t1 -c1 -M prepared',
2,
[],
@@ -371,7 +323,7 @@ SELECT 1 / (random() / 2)::int, :one::int, :two::int;
$node->append_conf('postgresql.conf', "log_min_duration_statement = 0");
$node->reload;
-pgbench(
+$node->pgbench(
'-n -t1 -c1 -M prepared',
2,
[],
@@ -394,7 +346,7 @@ like(
$log = undef;
# Check that bad parameters are reported during typinput phase of BIND
-pgbench(
+$node->pgbench(
'-n -t1 -c1 -M prepared',
2,
[],
@@ -418,7 +370,7 @@ $node->reload;
# test expressions
# command 1..3 and 23 depend on random seed which is used to call srandom.
-pgbench(
+$node->pgbench(
'--random-seed=5432 -t 1 -Dfoo=-10.1 -Dbla=false -Di=+3 -Dn=null -Dt=t -Df=of -Dd=1.0',
0,
[ qr{type: .*/001_pgbench_expressions}, qr{processed: 1/1} ],
@@ -653,7 +605,7 @@ $node->safe_psql('postgres',
my $seed = int(rand(1000000000));
for my $i (1, 2)
{
- pgbench(
+ $node->pgbench(
"--random-seed=$seed -t 1",
0,
[qr{processed: 1/1}],
@@ -693,7 +645,7 @@ ok($out =~ /\b$seed\|zipfian\|4\d\d\d\|2/,
$node->safe_psql('postgres', 'DROP TABLE seeded_random;');
# backslash commands
-pgbench(
+$node->pgbench(
'-t 1', 0,
[
qr{type: .*/001_pgbench_backslash_commands},
@@ -722,7 +674,7 @@ pgbench(
});
# working \gset
-pgbench(
+$node->pgbench(
'-t 1', 0,
[ qr{type: .*/001_pgbench_gset}, qr{processed: 1/1} ],
[
@@ -757,7 +709,7 @@ SELECT 0 AS i4, 4 AS i4 \gset
}
});
# \gset cannot accept more than one row, causing command to fail.
-pgbench(
+$node->pgbench(
'-t 1', 2,
[ qr{type: .*/001_pgbench_gset_two_rows}, qr{processed: 0/1} ],
[qr{expected one row, got 2\b}],
@@ -770,7 +722,7 @@ SELECT 5432 AS fail UNION SELECT 5433 ORDER BY 1 \gset
# working \aset
# Valid cases.
-pgbench(
+$node->pgbench(
'-t 1', 0,
[ qr{type: .*/001_pgbench_aset}, qr{processed: 1/1} ],
[ qr{command=3.: int 8\b}, qr{command=4.: int 7\b} ],
@@ -786,7 +738,7 @@ SELECT 8 AS i6 UNION SELECT 9 ORDER BY 1 DESC \aset
}
});
# Empty result set with \aset, causing command to fail.
-pgbench(
+$node->pgbench(
'-t 1', 2,
[ qr{type: .*/001_pgbench_aset_empty}, qr{processed: 0/1} ],
[
@@ -803,7 +755,7 @@ pgbench(
});
# Working \startpipeline
-pgbench(
+$node->pgbench(
'-t 1 -n -M extended',
0,
[ qr{type: .*/001_pgbench_pipeline}, qr{actually processed: 1/1} ],
@@ -819,7 +771,7 @@ pgbench(
});
# Working \startpipeline in prepared query mode
-pgbench(
+$node->pgbench(
'-t 1 -n -M prepared',
0,
[ qr{type: .*/001_pgbench_pipeline_prep}, qr{actually processed: 1/1} ],
@@ -835,7 +787,7 @@ pgbench(
});
# Try \startpipeline twice
-pgbench(
+$node->pgbench(
'-t 1 -n -M extended',
2,
[],
@@ -850,7 +802,7 @@ pgbench(
});
# Try to end a pipeline that hasn't started
-pgbench(
+$node->pgbench(
'-t 1 -n -M extended',
2,
[],
@@ -864,7 +816,7 @@ pgbench(
});
# Try \gset in pipeline mode
-pgbench(
+$node->pgbench(
'-t 1 -n -M extended',
2,
[],
@@ -1129,7 +1081,7 @@ for my $e (@errors)
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
- pgbench(
+ $node->pgbench(
'-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX'
. ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808'
. ($no_prepare ? '' : ' -M prepared'),
@@ -1141,14 +1093,14 @@ for my $e (@errors)
}
# throttling
-pgbench(
+$node->pgbench(
'-t 100 -S --rate=100000 --latency-limit=1000000 -c 2 -n -r',
0,
[ qr{processed: 200/200}, qr{builtin: select only} ],
[qr{^$}],
'pgbench throttling');
-pgbench(
+$node->pgbench(
# given the expected rate and the 2 ms tx duration, at most one is executed
'-t 10 --rate=100000 --latency-limit=1 -n -r',
@@ -1220,7 +1172,7 @@ sub check_pgbench_logs
my $bdir = $node->basedir;
# Run with sampling rate, 2 clients with 50 transactions each.
-pgbench(
+$node->pgbench(
"-n -S -t 50 -c 2 --log --sampling-rate=0.5", 0,
[ qr{select only}, qr{processed: 100/100} ], [qr{^$}],
'pgbench logs', undef,
@@ -1230,7 +1182,7 @@ check_pgbench_logs($bdir, '001_pgbench_log_2', 1, 8, 92,
qr{^[01] \d{1,2} \d+ \d \d+ \d+$});
# Run with different read-only option pattern, 1 client with 10 transactions.
-pgbench(
+$node->pgbench(
"-n -b select-only -t 10 -l", 0,
[ qr{select only}, qr{processed: 10/10} ], [qr{^$}],
'pgbench logs contents', undef,
diff --git a/src/include/utils/inval.h b/src/include/utils/inval.h
index 7706728..877e66c 100644
--- a/src/include/utils/inval.h
+++ b/src/include/utils/inval.h
@@ -62,6 +62,7 @@ extern void CacheRegisterRelcacheCallback(RelcacheCallbackFunction func,
extern void CallSyscacheCallbacks(int cacheid, uint32 hashvalue);
extern void InvalidateSystemCaches(void);
+extern void InvalidateSystemCachesExtended(bool debug_discard);
extern void LogLogicalInvalidations(void);
#endif /* INVAL_H */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index d2c1757..aa060ef 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -122,7 +122,7 @@ extern void RelationForgetRelation(Oid rid);
extern void RelationCacheInvalidateEntry(Oid relationId);
-extern void RelationCacheInvalidate(void);
+extern void RelationCacheInvalidate(bool debug_discard);
extern void RelationCloseSmgrByOid(Oid relationId);
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index c59da75..bfb338c 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1981,6 +1981,139 @@ sub interactive_psql
return $harness;
}
+# Common sub of pgbench-invoking interfaces. Makes any requested script files
+# and returns pgbench command-line options causing use of those files.
+sub _pgbench_make_files
+{
+ my ($self, $files) = @_;
+ my @file_opts;
+
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @file_opts, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ return @file_opts;
+}
+
+=pod
+
+=item $node->pgbench($opts, $stat, $out, $err, $name, $files, @args)
+
+Invoke B<pgbench>, with parameters and files.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $stat
+
+Expected exit status.
+
+=item $out
+
+Reference to a regexp list that must match stdout.
+
+=item $err
+
+Reference to a regexp list that must match stderr.
+
+=item $name
+
+Name of test for error messages.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=item @args
+
+Further raw options or arguments.
+
+=back
+
+=cut
+
+sub pgbench
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my ($self, $opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = (
+ 'pgbench',
+ split(/\s+/, $opts),
+ $self->_pgbench_make_files($files), @args);
+
+ $self->command_checks_all(\@cmd, $stat, $out, $err, $name);
+}
+
+=pod
+
+=item $node->background_pgbench($opts, $files, \$stdout, $timer) => harness
+
+Invoke B<pgbench> and return an IPC::Run harness object. The process's stdin
+is empty, and its stdout and stderr go to the $stdout scalar reference. This
+allows the caller to act on other parts of the system while B<pgbench> is
+running. Errors from B<pgbench> are the caller's problem.
+
+The specified timer object is attached to the harness, as well. It's caller's
+responsibility to select the timeout length, and to restart the timer after
+each command if the timeout is per-command.
+
+Be sure to "finish" the harness when done with it.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=back
+
+=cut
+
+sub background_pgbench
+{
+ my ($self, $opts, $files, $stdout, $timer) = @_;
+
+ my @cmd =
+ ('pgbench', split(/\s+/, $opts), $self->_pgbench_make_files($files));
+
+ local %ENV = $self->_get_env();
+
+ my $stdin = "";
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd, '<', \$stdin, '>', $stdout, $timer;
+
+ return $harness;
+}
+
=pod
=item $node->connect_ok($connstr, $test_name, %params)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cb5b5ec..1c7bac0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1093,6 +1093,7 @@ ImportForeignSchemaStmt
ImportForeignSchemaType
ImportForeignSchema_function
ImportQual
+InProgressEnt
IncludeWal
InclusionOpaque
IncrementVarSublevelsUp_context
prepared-transactions-cic-series202107-v15nm.patch (text/plain, 19.1K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
Fix CREATE INDEX CONCURRENTLY for the newest prepared transactions.
The purpose of commit 8a54e12a38d1545d249f1402f66c8cde2837d97c was to
fix this, and it sufficed when the PREPARE TRANSACTION completed before
the CIC looked for lock conflicts. Otherwise, things still broke. As
before, in a cluster having used CIC while having enabled prepared
transactions, queries that use the resulting index can silently fail to
find rows. It may be necessary to reindex to recover from past
occurrences; REINDEX CONCURRENTLY suffices. Fix this for future index
builds by making CIC wait for arbitrarily-recent prepared transactions
and for ordinary transactions that may yet PREPARE TRANSACTION. As part
of that, have PREPARE TRANSACTION transfer locks to its dummy PGPROC
before it calls ProcArrayClearTransaction(). Back-patch to 9.6 (all
supported versions).
Andrey Borodin, reviewed (in earlier versions) by Andres Freund.
Discussion: https://postgr.es/m/01824242-AA92-4FE9-9BA7-AEBAFFEA3D0C@yandex-team.ru
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
new file mode 100644
index 0000000..7c5c052
--- /dev/null
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -0,0 +1,187 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test CREATE INDEX CONCURRENTLY with concurrent prepared-xact modifications
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 6;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 180000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psql processes: $main_h for INSERTs and
+# $reindex_h for cuncurrent reindex. Also, we use non-background psql for
+# some COMMIT PREPARED statements.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(180);
+
+my $main_h =
+ $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint1
+);
+pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(180);
+my $reindex_h =
+ $node->background_psql('postgres', \$reindex_in, \$reindex_out,
+ $reindex_timer, on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint2
+);
+pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint3
+);
+pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check after overlapping 2PC');
+
+
+#
+# Server restart shall not change whether prepared xact blocks CIC
+#
+
+$node->safe_psql(
+ 'postgres', q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+PREPARE TRANSACTION 'spans_restart';
+BEGIN;
+CREATE TABLE unused ();
+PREPARE TRANSACTION 'persists_forever';
+));
+$node->restart;
+
+$reindex_in = '';
+$reindex_out = '';
+$reindex_timer = IPC::Run::timeout(180);
+$reindex_h =
+ $node->background_psql('postgres', \$reindex_in, \$reindex_out,
+ $reindex_timer, on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
+$reindex_h->finish;
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check after 2PC and restart');
+
+
+#
+# Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single
+# pgbench: CIC will deadlock with itself occasionally.
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench(
+ '--no-vacuum --client=1 --transactions=100',
+ {
+ '002_pgbench_concurrent_cic' => q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ \$pgbench_out,
+ $pgbench_timer);
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=100',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent INSERTs w/ 2PC',
+ {
+ '002_pgbench_concurrent_2pc' => q(
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ ),
+ '002_pgbench_concurrent_2pc_savepoint' => q(
+ BEGIN;
+ SAVEPOINT s1;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result =
+ ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->stop;
+done_testing();
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 2156de1..f6e7fa7 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,24 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ if (LocalTransactionIdIsValid(MyProc->lxid))
+ {
+ /* clone VXID, for TwoPhaseGetXidByVirtualXID() to find */
+ proc->lxid = MyProc->lxid;
+ proc->backendId = MyBackendId;
+ }
+ else
+ {
+ Assert(AmStartupProcess() || !IsPostmasterEnvironment);
+ /* GetLockConflicts() uses this to specify a wait on the XID */
+ proc->lxid = xid;
+ proc->backendId = InvalidBackendId;
+ }
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -847,6 +857,53 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
}
/*
+ * TwoPhaseGetXidByVirtualXID
+ * Lookup VXID among xacts prepared since last startup.
+ *
+ * (This won't find recovered xacts.) If more than one matches, return any
+ * and set "have_more" to true. To witness multiple matches, a single
+ * BackendId must consume 2^32 LXIDs, with no intervening database restart.
+ */
+TransactionId
+TwoPhaseGetXidByVirtualXID(VirtualTransactionId vxid,
+ bool *have_more)
+{
+ int i;
+ TransactionId result = InvalidTransactionId;
+
+ Assert(VirtualTransactionIdIsValid(vxid));
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc;
+ VirtualTransactionId proc_vxid;
+
+ if (!gxact->valid)
+ continue;
+ proc = &ProcGlobal->allProcs[gxact->pgprocno];
+ GET_VXID_FROM_PGPROC(proc_vxid, *proc);
+ if (VirtualTransactionIdEquals(vxid, proc_vxid))
+ {
+ /* Startup process sets proc->backendId to InvalidBackendId. */
+ Assert(!gxact->inredo);
+
+ if (result != InvalidTransactionId)
+ {
+ *have_more = true;
+ break;
+ }
+ result = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
+/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
*
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 4cc38f0..b7eed9f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2506,6 +2506,13 @@ PrepareTransaction(void)
XactLastRecEnd = 0;
/*
+ * Transfer our locks to a dummy PGPROC. This has to be done before
+ * ProcArrayClearTransaction(). Otherwise, a GetLockConflicts() would
+ * conclude "xact already committed or aborted" for our locks.
+ */
+ PostPrepare_Locks(xid);
+
+ /*
* Let others know about no transaction in progress by me. This has to be
* done *after* the prepared transaction has been marked valid, else
* someone may think it is unlocked and recyclable.
@@ -2544,7 +2551,6 @@ PrepareTransaction(void)
PostPrepare_MultiXact(xid);
- PostPrepare_Locks(xid);
PostPrepare_PredicateLocks(xid);
ResourceOwnerRelease(TopTransactionResourceOwner,
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index cdf2266..2db0424 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -871,9 +871,10 @@ XactLockTableWaitErrorCb(void *arg)
* To do this, obtain the current list of lockers, and wait on their VXIDs
* until they are finished.
*
- * Note we don't try to acquire the locks on the given locktags, only the VXIDs
- * of its lock holders; if somebody grabs a conflicting lock on the objects
- * after we obtained our initial list of lockers, we will not wait for them.
+ * Note we don't try to acquire the locks on the given locktags, only the
+ * VXIDs and XIDs of their lock holders; if somebody grabs a conflicting lock
+ * on the objects after we obtained our initial list of lockers, we will not
+ * wait for them.
*/
void
WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e..c25af7f 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -2900,8 +2900,12 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
* The result array is palloc'd and is terminated with an invalid VXID.
* *countp, if not null, is updated to the number of items set.
*
- * Of course, the result could be out of date by the time it's returned,
- * so use of this function has to be thought about carefully.
+ * Of course, the result could be out of date by the time it's returned, so
+ * use of this function has to be thought about carefully. Similarly, a
+ * PGPROC with no "lxid" will be considered non-conflicting regardless of any
+ * lock it holds. Existing callers don't care about a locker after that
+ * locker's pg_xact updates complete. CommitTransaction() clears "lxid" after
+ * pg_xact updates and before releasing locks.
*
* Note we never include the current xact's vxid in the result array,
* since an xact never blocks itself.
@@ -4530,36 +4534,79 @@ VirtualXactLockTableCleanup(void)
}
/*
+ * XactLockForVirtualXact
+ *
+ * If TransactionIdIsValid(xid), this is essentially XactLockTableWait(xid,
+ * NULL, NULL, XLTW_None) or ConditionalXactLockTableWait(xid). Unlike those
+ * functions, it assumes "xid" is never a subtransaction and that "xid" is
+ * prepared, committed, or aborted.
+ *
+ * If !TransactionIdIsValid(xid), this locks every prepared XID having been
+ * known as "vxid" before its PREPARE TRANSACTION.
+ */
+static bool
+XactLockForVirtualXact(VirtualTransactionId vxid,
+ TransactionId xid, bool wait)
+{
+ bool more = false;
+
+ /* There is no point to wait for 2PCs if you have no 2PCs. */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ do
+ {
+ LockAcquireResult lar;
+ LOCKTAG tag;
+
+ /* Clear state from previous iterations. */
+ if (more)
+ {
+ xid = InvalidTransactionId;
+ more = false;
+ }
+
+ /* If we have no xid, try to find one. */
+ if (!TransactionIdIsValid(xid))
+ xid = TwoPhaseGetXidByVirtualXID(vxid, &more);
+ if (!TransactionIdIsValid(xid))
+ {
+ Assert(!more);
+ return true;
+ }
+
+ /* Check or wait for XID completion. */
+ SET_LOCKTAG_TRANSACTION(tag, xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ return false;
+ LockRelease(&tag, ShareLock, false);
+ } while (more);
+
+ return true;
+}
+
+/*
* VirtualXactLock
*
- * If wait = true, wait until the given VXID has been released, and then
- * return true.
+ * If wait = true, wait as long as the given VXID or any XID acquired by the
+ * same transaction is still running. Then, return true.
*
- * If wait = false, just check whether the VXID is still running, and return
- * true or false.
+ * If wait = false, just check whether that VXID or one of those XIDs is still
+ * running, and return true or false.
*/
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
LOCKTAG tag;
PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
- if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ if (VirtualTransactionIdIsRecoveredPreparedXact(vxid))
+ /* no vxid lock; localTransactionId is a normal, locked XID */
+ return XactLockForVirtualXact(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4573,7 +4620,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return XactLockForVirtualXact(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4582,12 +4629,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
- /* If the transaction has ended, our work here is done. */
if (proc->backendId != vxid.backendId
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
+ /* VXID ended */
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return XactLockForVirtualXact(vxid, InvalidTransactionId, wait);
}
/*
@@ -4634,6 +4681,16 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
proc->fpVXIDLock = false;
}
+ /*
+ * If the proc has an XID now, we'll avoid a TwoPhaseGetXidByVirtualXID()
+ * search. The proc might have assigned this XID but not yet locked it,
+ * in which case the proc will lock this XID before releasing the VXID.
+ * The fpInfoLock critical section excludes VirtualXactLockTableCleanup(),
+ * so we won't save an XID of a different VXID. It doesn't matter whether
+ * we save this before or after setting up the primary lock table entry.
+ */
+ xid = proc->xid;
+
/* Done with proc->fpLockBits */
LWLockRelease(&proc->fpInfoLock);
@@ -4641,7 +4698,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return XactLockForVirtualXact(vxid, xid, wait);
}
/*
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index dd8586a..777fab4 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -65,6 +65,20 @@
* (XXX is it worth testing likewise for duplicate catcache flush entries?
* Probably not.)
*
+ * Many subsystems own higher-level caches that depend on relcache and/or
+ * catcache, and they register callbacks here to invalidate their caches.
+ * While building a higher-level cache entry, a backend may receive a
+ * callback for the being-built entry or one of its dependencies. This
+ * implies the new higher-level entry would be born stale, and it might
+ * remain stale for the life of the backend. Many caches do not prevent
+ * that. They rely on DDL for can't-miss catalog changes taking
+ * AccessExclusiveLock on suitable objects. (For a change made with less
+ * locking, backends might never read the change.) The relation cache,
+ * however, needs to reflect changes from CREATE INDEX CONCURRENTLY no later
+ * than the beginning of the next transaction. Hence, when a relevant
+ * invalidation callback arrives during a build, relcache.c reattempts that
+ * build. Caches with similar needs could do likewise.
+ *
* If a relcache flush is issued for a system relation that we preload
* from the relcache init file, we must also delete the init file so that
* it will be rebuilt during the next backend restart. The actual work of
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index e27e1a8..2d75842 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -34,6 +34,8 @@ extern void TwoPhaseShmemInit(void);
extern void AtAbort_Twophase(void);
extern void PostPrepare_Twophase(void);
+extern TransactionId TwoPhaseGetXidByVirtualXID(VirtualTransactionId vxid,
+ bool *have_more);
extern PGPROC *TwoPhaseGetDummyProc(TransactionId xid, bool lock_held);
extern BackendId TwoPhaseGetDummyBackendId(TransactionId xid, bool lock_held);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index 9b2a421..a5286fa 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -47,10 +47,11 @@ extern bool Debug_deadlocks;
/*
* Top-level transactions are identified by VirtualTransactionIDs comprising
- * PGPROC fields backendId and lxid. For prepared transactions, the
- * LocalTransactionId is an ordinary XID. These are guaranteed unique over
- * the short term, but will be reused after a database restart or XID
- * wraparound; hence they should never be stored on disk.
+ * PGPROC fields backendId and lxid. For recovered prepared transactions, the
+ * LocalTransactionId is an ordinary XID; LOCKTAG_VIRTUALTRANSACTION never
+ * refers to that kind. These are guaranteed unique over the short term, but
+ * will be reused after a database restart or XID wraparound; hence they
+ * should never be stored on disk.
*
* Note that struct VirtualTransactionId can not be assumed to be atomically
* assignable as a whole. However, type LocalTransactionId is assumed to
@@ -70,7 +71,7 @@ typedef struct
#define LocalTransactionIdIsValid(lxid) ((lxid) != InvalidLocalTransactionId)
#define VirtualTransactionIdIsValid(vxid) \
(LocalTransactionIdIsValid((vxid).localTransactionId))
-#define VirtualTransactionIdIsPreparedXact(vxid) \
+#define VirtualTransactionIdIsRecoveredPreparedXact(vxid) \
((vxid).backendId == InvalidBackendId)
#define VirtualTransactionIdEquals(vxid1, vxid2) \
((vxid1).backendId == (vxid2).backendId && \
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-18 13:23 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-10-18 13:23 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 17 окт. 2021 г., в 20:12, Noah Misch <noah@leadboat.com> написал(а):
>
> I think the attached version is ready for commit. Notable differences
> vs. v14:
Few notes:
1. Maybe an Assert(in_progress_list_maxlen) when in_progress_list_maxlen is used?
2.
-#define VirtualTransactionIdIsPreparedXact(vxid) \
+#define VirtualTransactionIdIsRecoveredPreparedXact(vxid) \
This is a very neat transition. Yes, the function argument will always be a xid only for recovered transactions. Maybe add a comment here that this function is expected to be used only for results of GetLockConflicts()?
> One thing not done here is to change the tests to use CREATE INDEX
> CONCURRENTLY instead of REINDEX CONCURRENTLY, so they're back-patchable to v11
> and earlier. I may do that before pushing, or I may just omit the tests from
> older branches.
The tests refactors PostgresNode.pm and some tests. Back-patching this would be quite invasive.
But swapping every "REINDEX INDEX CONCURRENTLY idx;" with
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx on tbl(i);
works.
> <inval-build-race-v1.patch><prepared-transactions-cic-series202107-v15nm.patch>
I've checked that this patches work for some time on my machines. I do not observe failures.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-10-19 03:02 ` Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-10-19 03:02 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Mon, Oct 18, 2021 at 06:23:05PM +0500, Andrey Borodin wrote:
> > 17 окт. 2021 г., в 20:12, Noah Misch <noah@leadboat.com> написал(а):
> > I think the attached version is ready for commit. Notable differences
> > vs. v14:
> Few notes:
>
> 1. Maybe an Assert(in_progress_list_maxlen) when in_progress_list_maxlen is used?
RelationCacheInitialize() initializes both in_progress_list_maxlen and the
RelationIdCache hash table, and any ERROR there is promoted to FATAL. Without
the hash table, nothing good can happen in relcache. Hence, I think such an
assert is excessive.
> 2.
> -#define VirtualTransactionIdIsPreparedXact(vxid) \
> +#define VirtualTransactionIdIsRecoveredPreparedXact(vxid) \
>
> This is a very neat transition. Yes, the function argument will always be a xid only for recovered transactions. Maybe add a comment here that this function is expected to be used only for results of GetLockConflicts()?
One can use it on any VirtualTransactionId, though some sources only return
values for which this returns false. It can return true for the result of
GET_VXID_FROM_PGPROC(). I think it can return true for the result of
GetCurrentVirtualXIDs(limitXmin = InvalidTransactionId), but core code doesn't
make such a call.
> > One thing not done here is to change the tests to use CREATE INDEX
> > CONCURRENTLY instead of REINDEX CONCURRENTLY, so they're back-patchable to v11
> > and earlier. I may do that before pushing, or I may just omit the tests from
> > older branches.
>
> The tests refactors PostgresNode.pm and some tests. Back-patching this would be quite invasive.
That's fine with me. Back-patching a fix without its tests is riskier than
back-patching test infrastructure changes.
> But swapping every "REINDEX INDEX CONCURRENTLY idx;" with
> DROP INDEX CONCURRENTLY idx;
> CREATE INDEX CONCURRENTLY idx on tbl(i);
> works.
>
> > <inval-build-race-v1.patch><prepared-transactions-cic-series202107-v15nm.patch>
> I've checked that this patches work for some time on my machines. I do not observe failures.
Good. Thanks for checking.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-24 05:00 ` Noah Misch <noah@leadboat.com>
2021-10-24 11:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 2 replies; 107+ messages in thread
From: Noah Misch @ 2021-10-24 05:00 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Mon, Oct 18, 2021 at 08:02:12PM -0700, Noah Misch wrote:
> On Mon, Oct 18, 2021 at 06:23:05PM +0500, Andrey Borodin wrote:
> > > 17 окт. 2021 г., в 20:12, Noah Misch <noah@leadboat.com> написал(а):
> > > I think the attached version is ready for commit. Notable differences
> > > vs. v14:
Pushed. Buildfarm member conchuela (DragonFly BSD 6.0) has gotten multiple
"IPC::Run: timeout on timer" in the new tests. No other animal has.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2021-10-24%2003%3A05%3A09
is an example run. The pgbench queries finished quickly, but the
$pgbench_h->finish() apparently timed out after 180s. I guess this would be
consistent with pgbench blocking in write(), waiting for something to empty a
pipe buffer so it can write more. I thought finish() will drain any incoming
I/O, though. This phenomenon has been appearing regularly via
src/test/recovery/t/017_shm.pl[1], so this thread doesn't have a duty to
resolve it. A stack trace of the stuck pgbench should be informative, though.
Compared to my last post, the push included two more test changes. I removed
sleeps from a test. They could add significant time on a system with coarse
sleep granularity. This did not change test sensitivity on my system.
Second, I changed background_pgbench to include stderr lines in $stdout, as it
had documented. This becomes important during the back-patch to v11, where
server errors don't cause a nonzero pgbench exit status. background_psql
still has the same bug, and I can fix it later. (The background_psql version
of the bug is not affecting current usage.)
FYI, the non-2PC test is less sensitive in older branches. It reproduces
master's bug in 25-50% of runs, but it took about six minutes on v11 and v12.
> > > One thing not done here is to change the tests to use CREATE INDEX
> > > CONCURRENTLY instead of REINDEX CONCURRENTLY, so they're back-patchable to v11
> > > and earlier. I may do that before pushing, or I may just omit the tests from
> > > older branches.
> >
> > The tests refactors PostgresNode.pm and some tests. Back-patching this would be quite invasive.
>
> That's fine with me. Back-patching a fix without its tests is riskier than
> back-patching test infrastructure changes.
Back-patching the tests did end up tricky, for other reasons. Before v12
(d3c09b9), a TAP suite in a pgxs module wouldn't run during check-world.
Before v11 (7f563c0), amcheck lacks the heapallindexed feature that the tests
rely on. Hence, for v11, v10, and v9.6, I used a plpgsql implementation of
the heapallindexed check, and I moved the tests to src/bin/pgbench.
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2021-10-19%2012%3A58%3A08
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-24 11:45 ` Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 107+ messages in thread
From: Andrey Borodin @ 2021-10-24 11:45 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 24 окт. 2021 г., в 08:00, Noah Misch <noah@leadboat.com> написал(а):
>
> On Mon, Oct 18, 2021 at 08:02:12PM -0700, Noah Misch wrote:
>> On Mon, Oct 18, 2021 at 06:23:05PM +0500, Andrey Borodin wrote:
>>>> 17 окт. 2021 г., в 20:12, Noah Misch <noah@leadboat.com> написал(а):
>>>> I think the attached version is ready for commit. Notable differences
>>>> vs. v14:
>
> Pushed.
Wow, that's great! Thank you!
> Buildfarm member conchuela (DragonFly BSD 6.0) has gotten multiple
> "IPC::Run: timeout on timer" in the new tests. No other animal has.
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2021-10-24%2003%3A05%3A09
> is an example run. The pgbench queries finished quickly, but the
> $pgbench_h->finish() apparently timed out after 180s. I guess this would be
> consistent with pgbench blocking in write(), waiting for something to empty a
> pipe buffer so it can write more. I thought finish() will drain any incoming
> I/O, though. This phenomenon has been appearing regularly via
> src/test/recovery/t/017_shm.pl[1], so this thread doesn't have a duty to
> resolve it. A stack trace of the stuck pgbench should be informative, though.
Some thoughts:
0. I doubt that psql\pgbench is stuck in these failures.
1. All observed similar failures seem to be related to finish() sub of IPC::Run harness
2. Finish must pump any pending data from process [0]. But it can hang if process is waiting for something.
3. There is reported bug of finish [1]. But the description is slightly different.
>
> Compared to my last post, the push included two more test changes. I removed
> sleeps from a test. They could add significant time on a system with coarse
> sleep granularity. This did not change test sensitivity on my system.
> Second, I changed background_pgbench to include stderr lines in $stdout, as it
> had documented. This becomes important during the back-patch to v11, where
> server errors don't cause a nonzero pgbench exit status. background_psql
> still has the same bug, and I can fix it later. (The background_psql version
> of the bug is not affecting current usage.)
>
> FYI, the non-2PC test is less sensitive in older branches. It reproduces
> master's bug in 25-50% of runs, but it took about six minutes on v11 and v12.
It seem like loading Relation Descr to relcache becomes more expensive?
>>>> One thing not done here is to change the tests to use CREATE INDEX
>>>> CONCURRENTLY instead of REINDEX CONCURRENTLY, so they're back-patchable to v11
>>>> and earlier. I may do that before pushing, or I may just omit the tests from
>>>> older branches.
>>>
>>> The tests refactors PostgresNode.pm and some tests. Back-patching this would be quite invasive.
>>
>> That's fine with me. Back-patching a fix without its tests is riskier than
>> back-patching test infrastructure changes.
>
> Back-patching the tests did end up tricky, for other reasons. Before v12
> (d3c09b9), a TAP suite in a pgxs module wouldn't run during check-world.
> Before v11 (7f563c0), amcheck lacks the heapallindexed feature that the tests
> rely on. Hence, for v11, v10, and v9.6, I used a plpgsql implementation of
> the heapallindexed check, and I moved the tests to src/bin/pgbench.
Cool!
Thanks!
Best regards, Andrey Borodin.
[0] https://metacpan.org/dist/IPC-Run/source/lib/IPC/Run.pm#L3481
[1] https://github.com/toddr/IPC-Run/issues/57
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-24 23:35 ` Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-10-24 23:35 UTC (permalink / raw)
To: cm@enterprisedb.com, Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Oct 23, 2021 at 10:00:28PM -0700, Noah Misch wrote:
> Pushed.
cm@enterprisedb.com, could you post a stack trace from buildfarm member
gharial failing in "make check"? I'm looking for a trace from a SIGSEGV like
those seen today:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2016%3A19%3A05
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2020%3A38%3A39
Those SIGSEGV are all in ALTER TABLE. My commits from this thread probably
broke something, but I've run out of ideas, and my hpux boxes died.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=kittiwake&dt=2021-10-24%2012%3A01%3A10
got an interesting v9.6 failure, unrelated to the gharial SIGSEGV failures:
2021-10-24 14:25:29.263 CEST [34569:175] pgbench ERROR: could not read two-phase state from xlog at 0/158F4E0
2021-10-24 14:25:29.263 CEST [34569:176] pgbench STATEMENT: COMMIT PREPARED 'c1';
I don't see this thread's commits changing code paths relevant to that
failure, so I currently think this one is a new test showing an old bug.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-28 14:04 ` Noah Misch <noah@leadboat.com>
2021-10-28 20:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
0 siblings, 2 replies; 107+ messages in thread
From: Noah Misch @ 2021-10-28 14:04 UTC (permalink / raw)
To: cm@enterprisedb.com; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>
On Sun, Oct 24, 2021 at 04:35:02PM -0700, Noah Misch wrote:
> On Sat, Oct 23, 2021 at 10:00:28PM -0700, Noah Misch wrote:
> > Pushed.
>
> cm@enterprisedb.com, could you post a stack trace from buildfarm member
> gharial failing in "make check"? I'm looking for a trace from a SIGSEGV like
> those seen today:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2016%3A19%3A05
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2020%3A38%3A39
>
> Those SIGSEGV are all in ALTER TABLE. My commits from this thread probably
> broke something, but I've run out of ideas, and my hpux boxes died.
cm@enterprisedb.com, is this still in your queue, or not? I am somewhat
concerned that the 2021-11-11 releases will contain a regression if the
gharial failures remain undiagnosed. I don't know how to diagnose them
without your help. (I currently can't justify reverting the suspect bug fix
on the basis of two master-branch-only failures in one buildfarm member.)
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-28 20:59 ` Thomas Munro <thomas.munro@gmail.com>
2021-10-29 02:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 107+ messages in thread
From: Thomas Munro @ 2021-10-28 20:59 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: cm@enterprisedb.com, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>
On Fri, Oct 29, 2021 at 3:05 AM Noah Misch <noah@leadboat.com> wrote:
> > Those SIGSEGV are all in ALTER TABLE. My commits from this thread probably
> > broke something, but I've run out of ideas, and my hpux boxes died.
I recently managed to scrounge up an account in another generous
project's operating system graveyard, and could possibly try to repro
that or put you in touch. On first attempt just now I couldn't build
because of weird problems with ifaddr.c's need for structs in
<net/if.h> that seem to be kernel/internal-only that I unfortunately
won't have time to debug until next week (I've successfully built here
before so I must be doing something stupid).
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 20:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
@ 2021-10-29 02:49 ` Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 107+ messages in thread
From: Noah Misch @ 2021-10-29 02:49 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: cm@enterprisedb.com, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>
On Fri, Oct 29, 2021 at 09:59:14AM +1300, Thomas Munro wrote:
> On Fri, Oct 29, 2021 at 3:05 AM Noah Misch <noah@leadboat.com> wrote:
> > > Those SIGSEGV are all in ALTER TABLE. My commits from this thread probably
> > > broke something, but I've run out of ideas, and my hpux boxes died.
>
> I recently managed to scrounge up an account in another generous
> project's operating system graveyard, and could possibly try to repro
> that or put you in touch. On first attempt just now I couldn't build
> because of weird problems with ifaddr.c's need for structs in
> <net/if.h> that seem to be kernel/internal-only that I unfortunately
> won't have time to debug until next week (I've successfully built here
> before so I must be doing something stupid).
Thanks. Either would help. If you can put me in touch this week, let's try
that since there's little time left before release.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-30 06:09 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 107+ messages in thread
From: Sandeep Thakkar @ 2021-10-30 06:09 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>
Hi Noah,
On Thu, Oct 28, 2021 at 7:35 PM Noah Misch <noah@leadboat.com> wrote:
> On Sun, Oct 24, 2021 at 04:35:02PM -0700, Noah Misch wrote:
> > On Sat, Oct 23, 2021 at 10:00:28PM -0700, Noah Misch wrote:
> > > Pushed.
> >
> > cm@enterprisedb.com, could you post a stack trace from buildfarm member
> > gharial failing in "make check"? I'm looking for a trace from a SIGSEGV
> like
> > those seen today:
> >
> >
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2016%3A19%3A05
> >
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2020%3A38%3A39
> >
> > Those SIGSEGV are all in ALTER TABLE. My commits from this thread
> probably
> > broke something, but I've run out of ideas, and my hpux boxes died.
>
> cm@enterprisedb.com, is this still in your queue, or not? I am somewhat
> concerned that the 2021-11-11 releases will contain a regression if the
> gharial failures remain undiagnosed. I don't know how to diagnose them
> without your help. (I currently can't justify reverting the suspect bug
> fix
> on the basis of two master-branch-only failures in one buildfarm member.)
>
> Sorry for responding late as this email was missed somehow :-( I checked
the reports on the dashboard and there was a failure then but the reports
after 25th Oct looks fine. The back branches all look fine as well.
https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=gharial&br=HEAD
Please let me know if I missed something
--
Sandeep Thakkar
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
@ 2021-10-30 07:26 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-10-30 07:26 UTC (permalink / raw)
To: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; +Cc: Noah Misch <noah@leadboat.com>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 30 окт. 2021 г., в 11:09, Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> написал(а):
>
> On Thu, Oct 28, 2021 at 7:35 PM Noah Misch <noah@leadboat.com> wrote:
> On Sun, Oct 24, 2021 at 04:35:02PM -0700, Noah Misch wrote:
> > On Sat, Oct 23, 2021 at 10:00:28PM -0700, Noah Misch wrote:
> > > Pushed.
> >
> > cm@enterprisedb.com, could you post a stack trace from buildfarm member
> > gharial failing in "make check"? I'm looking for a trace from a SIGSEGV like
> > those seen today:
> >
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2016%3A19%3A05
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2020%3A38%3A39
> >
> > Those SIGSEGV are all in ALTER TABLE. My commits from this thread probably
> > broke something, but I've run out of ideas, and my hpux boxes died.
>
> cm@enterprisedb.com, is this still in your queue, or not? I am somewhat
> concerned that the 2021-11-11 releases will contain a regression if the
> gharial failures remain undiagnosed. I don't know how to diagnose them
> without your help. (I currently can't justify reverting the suspect bug fix
> on the basis of two master-branch-only failures in one buildfarm member.)
>
> Sorry for responding late as this email was missed somehow :-( I checked the reports on the dashboard and there was a failure then but the reports after 25th Oct looks fine. The back branches all look fine as well. https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=gharial&br=HEAD
>
> Please let me know if I missed something
Hi Sandeep, thank you for the response!
Some failed runs on this animal indicate SegFault in places changed by bugfix relevant to this thread. Fails were observed only on HEAD, but this may be a result of concurrent nature of possible new bug. If there is a new bug - this would affect upcoming point release. But we do not know for sure. Other animals show no signs of any related SegFault.
Can you please run make check phase on this animal on HEAD multiple time (preferably ~hundred times)? If some of this runs fail it's vital to collect backtraces of run.
Many thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-10-30 07:59 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Sandeep Thakkar @ 2021-10-30 07:59 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Oct 30, 2021 at 12:56 PM Andrey Borodin <x4mmm@yandex-team.ru>
wrote:
>
>
> > 30 окт. 2021 г., в 11:09, Sandeep Thakkar <
> sandeep.thakkar@enterprisedb.com> написал(а):
> >
> > On Thu, Oct 28, 2021 at 7:35 PM Noah Misch <noah@leadboat.com> wrote:
> > On Sun, Oct 24, 2021 at 04:35:02PM -0700, Noah Misch wrote:
> > > On Sat, Oct 23, 2021 at 10:00:28PM -0700, Noah Misch wrote:
> > > > Pushed.
> > >
> > > cm@enterprisedb.com, could you post a stack trace from buildfarm
> member
> > > gharial failing in "make check"? I'm looking for a trace from a
> SIGSEGV like
> > > those seen today:
> > >
> > >
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2016%3A19%3A05
> > >
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2020%3A38%3A39
> > >
> > > Those SIGSEGV are all in ALTER TABLE. My commits from this thread
> probably
> > > broke something, but I've run out of ideas, and my hpux boxes died.
> >
> > cm@enterprisedb.com, is this still in your queue, or not? I am somewhat
> > concerned that the 2021-11-11 releases will contain a regression if the
> > gharial failures remain undiagnosed. I don't know how to diagnose them
> > without your help. (I currently can't justify reverting the suspect bug
> fix
> > on the basis of two master-branch-only failures in one buildfarm member.)
> >
> > Sorry for responding late as this email was missed somehow :-( I
> checked the reports on the dashboard and there was a failure then but the
> reports after 25th Oct looks fine. The back branches all look fine as well.
> https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=gharial&br=HEAD
> >
> > Please let me know if I missed something
>
> Hi Sandeep, thank you for the response!
>
> Some failed runs on this animal indicate SegFault in places changed by
> bugfix relevant to this thread. Fails were observed only on HEAD, but this
> may be a result of concurrent nature of possible new bug. If there is a new
> bug - this would affect upcoming point release. But we do not know for
> sure. Other animals show no signs of any related SegFault.
> Can you please run make check phase on this animal on HEAD multiple time
> (preferably ~hundred times)? If some of this runs fail it's vital to
> collect backtraces of run.
>
> Hi Andrew,
OK I got it now. I've scheduled "./run_build.pl HEAD" on this animal to run
multiple times in a day. Unfortunately, I can't run it ~100 times because
it's a legacy (slow) server and also we run another animal (anole - with a
different compiler) on the same server. Depending on the time every run on
HEAD takes, I'll increase the frequency. Hope this helps.
> Many thanks!
>
> Best regards, Andrey Borodin.
--
Sandeep Thakkar
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
@ 2021-10-30 08:14 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Sandeep Thakkar @ 2021-10-30 08:14 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Oct 30, 2021 at 1:29 PM Sandeep Thakkar <
sandeep.thakkar@enterprisedb.com> wrote:
>
>
> On Sat, Oct 30, 2021 at 12:56 PM Andrey Borodin <x4mmm@yandex-team.ru>
> wrote:
>
>>
>>
>> > 30 окт. 2021 г., в 11:09, Sandeep Thakkar <
>> sandeep.thakkar@enterprisedb.com> написал(а):
>> >
>> > On Thu, Oct 28, 2021 at 7:35 PM Noah Misch <noah@leadboat.com> wrote:
>> > On Sun, Oct 24, 2021 at 04:35:02PM -0700, Noah Misch wrote:
>> > > On Sat, Oct 23, 2021 at 10:00:28PM -0700, Noah Misch wrote:
>> > > > Pushed.
>> > >
>> > > cm@enterprisedb.com, could you post a stack trace from buildfarm
>> member
>> > > gharial failing in "make check"? I'm looking for a trace from a
>> SIGSEGV like
>> > > those seen today:
>> > >
>> > >
>> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2016%3A19%3A05
>> > >
>> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial&dt=2021-10-24%2020%3A38%3A39
>> > >
>> > > Those SIGSEGV are all in ALTER TABLE. My commits from this thread
>> probably
>> > > broke something, but I've run out of ideas, and my hpux boxes died.
>> >
>> > cm@enterprisedb.com, is this still in your queue, or not? I am
>> somewhat
>> > concerned that the 2021-11-11 releases will contain a regression if the
>> > gharial failures remain undiagnosed. I don't know how to diagnose them
>> > without your help. (I currently can't justify reverting the suspect
>> bug fix
>> > on the basis of two master-branch-only failures in one buildfarm
>> member.)
>> >
>> > Sorry for responding late as this email was missed somehow :-( I
>> checked the reports on the dashboard and there was a failure then but the
>> reports after 25th Oct looks fine. The back branches all look fine as well.
>> https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=gharial&br=HEAD
>> >
>> > Please let me know if I missed something
>>
>> Hi Sandeep, thank you for the response!
>>
>> Some failed runs on this animal indicate SegFault in places changed by
>> bugfix relevant to this thread. Fails were observed only on HEAD, but this
>> may be a result of concurrent nature of possible new bug. If there is a new
>> bug - this would affect upcoming point release. But we do not know for
>> sure. Other animals show no signs of any related SegFault.
>> Can you please run make check phase on this animal on HEAD multiple time
>> (preferably ~hundred times)? If some of this runs fail it's vital to
>> collect backtraces of run.
>>
>> Hi Andrew,
>
> OK I got it now. I've scheduled "./run_build.pl HEAD" on this animal to
> run multiple times in a day. Unfortunately, I can't run it ~100 times
> because it's a legacy (slow) server and also we run another animal (anole -
> with a different compiler) on the same server. Depending on the time every
> run on HEAD takes, I'll increase the frequency. Hope this helps.
>
> I've used "--force" option so that it ignores the last running status.
./run_build.pl --verbose --force HEAD
>
>
>> Many thanks!
>>
>> Best regards, Andrey Borodin.
>
>
>
> --
> Sandeep Thakkar
>
>
>
--
Sandeep Thakkar
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
@ 2021-10-30 19:19 ` Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-10-30 19:19 UTC (permalink / raw)
To: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Oct 30, 2021 at 01:44:56PM +0530, Sandeep Thakkar wrote:
> On Sat, Oct 30, 2021 at 1:29 PM Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> wrote:
> > On Sat, Oct 30, 2021 at 12:56 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> >> Some failed runs on this animal indicate SegFault in places changed by
> >> bugfix relevant to this thread. Fails were observed only on HEAD, but this
> >> may be a result of concurrent nature of possible new bug. If there is a new
> >> bug - this would affect upcoming point release. But we do not know for
> >> sure. Other animals show no signs of any related SegFault.
> >> Can you please run make check phase on this animal on HEAD multiple time
> >> (preferably ~hundred times)? If some of this runs fail it's vital to
> >> collect backtraces of run.
> >
> > OK I got it now. I've scheduled "./run_build.pl HEAD" on this animal to
> > run multiple times in a day. Unfortunately, I can't run it ~100 times
> > because it's a legacy (slow) server and also we run another animal (anole -
> > with a different compiler) on the same server. Depending on the time every
> > run on HEAD takes, I'll increase the frequency. Hope this helps.
>
> I've used "--force" option so that it ignores the last running status.
> ./run_build.pl --verbose --force HEAD
Thanks, but I don't think that gets us closer to having a stack trace for the
SIGSEGV. If some run gets a SIGSEGV, the core dump will get unlinked. I'd
disable both buildfarm member cron jobs and then run this script (derived from
gharial's buildfarm configuration):
git clone https://git.postgresql.org/git/postgresql.git
cd postgresql
git checkout 70bef49
export LD_LIBRARY_PATH='/opt/uuid-1.6.2/inst/lib:/opt/packages/uuid-1.6.2/inst/lib:/opt/packages/krb5-1.11.3/inst/lib/hpux64:/opt/packages/libmemcached-0.46/inst/lib/hpux64:/opt/packages/libevent-2.0.10/inst/lib/hpux64:/opt/packages/expat-2.0.1/inst/lib/hpux64:/opt/packages/gdbm-1.8.3/inst/lib/hpux64:/opt/packages/openldap-2.4.24/inst/lib/hpux64:/opt/packages/proj-4.7.0/inst/lib:/opt/packages/geos-3.2.2/inst/lib:/opt/packages/db-5.1.19/inst/lib/hpux64:/opt/packages/freetype-2.4.4/inst/lib/hpux64:/opt/packages/tcltk-8.5.9/inst/lib/hpux64:/opt/packages/openssl-1.0.0d/inst/lib/hpux64:/opt/packages/editline-2.9/inst/lib/hpux64:/opt/packages/mutt-1.5.21/inst/lib/hpux64:/opt/packages/libidn-1.20/inst/lib/hpux64:/opt/packages/libxslt-1.1.26/inst/lib/hpux64:/opt/packages/libgcrypt-1.4.6/inst/lib/hpux64:/opt/packages/libgpg_error-1.10/inst/lib/hpux64:/opt/packages/libxml2-2.7.8/inst/lib/hpux64:/opt/packages/zlib-1.2.5/inst/lib/hpux64:/opt/packages/grep-2.7/inst/lib/hpux64:/opt/packages/pcre-8.12/inst/lib/hpux64:/opt/packages/ncurses-5.8/inst/lib/hpux64:/opt/packages/termcap-1.3.1/inst/lib/hpux64:/opt/packages/gettext-0.18.1.1/inst/lib/hpux64:/opt/packages/libiconv-1.13.1/inst/lib/hpux64:/opt/packages/sdk-10.2.0.5.0-hpux-ia64/instantclient_10_2/lib'
cat >temp.conf <<EOF
log_line_prefix = '%m [%p:%l] %q%a '
log_connections = 'true'
log_disconnections = 'true'
log_statement = 'all'
fsync = off
force_parallel_mode = regress
EOF
./configure --enable-cassert --enable-debug --with-perl --without-readline \
--with-libxml --with-libxslt --with-libs=/opt/packages/zlib-1.2.5/inst/lib/hpux64:/opt/packages/libxslt-1.1.26/inst/lib/hpux64:/opt/packages/libxml2-2.7.8/inst/lib/hpux64 \
--with-includes=/opt/packages/zlib-1.2.5/inst/include:/opt/packages/libxslt-1.1.26/inst/include:/opt/packages/libxml2-2.7.8/inst/include \
--with-pgport=5678 CFLAGS=-mlp64 CC=gcc
make
while make check TEMP_CONFIG=$PWD/temp.conf NO_LOCALE=1; do :; done
# When the loop stops, use the core file to make a stack trace. If it runs
# for a week without stopping, give up.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-10-31 07:31 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 107+ messages in thread
From: Sandeep Thakkar @ 2021-10-31 07:31 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Semab Tariq <semab.tariq@enterprisedb.com>
Hi Noah,
On Sun, Oct 31, 2021 at 12:49 AM Noah Misch <noah@leadboat.com> wrote:
> On Sat, Oct 30, 2021 at 01:44:56PM +0530, Sandeep Thakkar wrote:
> > On Sat, Oct 30, 2021 at 1:29 PM Sandeep Thakkar <
> sandeep.thakkar@enterprisedb.com> wrote:
> > > On Sat, Oct 30, 2021 at 12:56 PM Andrey Borodin <x4mmm@yandex-team.ru>
> wrote:
> > >> Some failed runs on this animal indicate SegFault in places changed by
> > >> bugfix relevant to this thread. Fails were observed only on HEAD, but
> this
> > >> may be a result of concurrent nature of possible new bug. If there is
> a new
> > >> bug - this would affect upcoming point release. But we do not know for
> > >> sure. Other animals show no signs of any related SegFault.
> > >> Can you please run make check phase on this animal on HEAD multiple
> time
> > >> (preferably ~hundred times)? If some of this runs fail it's vital to
> > >> collect backtraces of run.
> > >
> > > OK I got it now. I've scheduled "./run_build.pl HEAD" on this animal
> to
> > > run multiple times in a day. Unfortunately, I can't run it ~100 times
> > > because it's a legacy (slow) server and also we run another animal
> (anole -
> > > with a different compiler) on the same server. Depending on the time
> every
> > > run on HEAD takes, I'll increase the frequency. Hope this helps.
> >
> > I've used "--force" option so that it ignores the last running status.
> > ./run_build.pl --verbose --force HEAD
>
> Thanks, but I don't think that gets us closer to having a stack trace for
> the
> SIGSEGV. If some run gets a SIGSEGV, the core dump will get unlinked. I'd
> disable both buildfarm member cron jobs and then run this script (derived
> from
> gharial's buildfarm configuration):
>
> git clone https://git.postgresql.org/git/postgresql.git
> cd postgresql
> git checkout 70bef49
> export
> LD_LIBRARY_PATH='/opt/uuid-1.6.2/inst/lib:/opt/packages/uuid-1.6.2/inst/lib:/opt/packages/krb5-1.11.3/inst/lib/hpux64:/opt/packages/libmemcached-0.46/inst/lib/hpux64:/opt/packages/libevent-2.0.10/inst/lib/hpux64:/opt/packages/expat-2.0.1/inst/lib/hpux64:/opt/packages/gdbm-1.8.3/inst/lib/hpux64:/opt/packages/openldap-2.4.24/inst/lib/hpux64:/opt/packages/proj-4.7.0/inst/lib:/opt/packages/geos-3.2.2/inst/lib:/opt/packages/db-5.1.19/inst/lib/hpux64:/opt/packages/freetype-2.4.4/inst/lib/hpux64:/opt/packages/tcltk-8.5.9/inst/lib/hpux64:/opt/packages/openssl-1.0.0d/inst/lib/hpux64:/opt/packages/editline-2.9/inst/lib/hpux64:/opt/packages/mutt-1.5.21/inst/lib/hpux64:/opt/packages/libidn-1.20/inst/lib/hpux64:/opt/packages/libxslt-1.1.26/inst/lib/hpux64:/opt/packages/libgcrypt-1.4.6/inst/lib/hpux64:/opt/packages/libgpg_error-1.10/inst/lib/hpux64:/opt/packages/libxml2-2.7.8/inst/lib/hpux64:/opt/packages/zlib-1.2.5/inst/lib/hpux64:/opt/packages/grep-2.7/inst/lib/hpux64:/opt/packages/pcre-8.12/inst/lib/hpux64:/opt/packages/ncurses-5.8/inst/lib/hpux64:/opt/packages/termcap-1.3.1/inst/lib/hpux64:/opt/packages/gettext-0.18.1.1/inst/lib/hpux64:/opt/packages/libiconv-1.13.1/inst/lib/hpux64:/opt/packages/sdk-10.2.0.5.0-hpux-ia64/instantclient_10_2/lib'
> cat >temp.conf <<EOF
> log_line_prefix = '%m [%p:%l] %q%a '
> log_connections = 'true'
> log_disconnections = 'true'
> log_statement = 'all'
> fsync = off
> force_parallel_mode = regress
> EOF
> ./configure --enable-cassert --enable-debug --with-perl --without-readline
> \
> --with-libxml --with-libxslt
> --with-libs=/opt/packages/zlib-1.2.5/inst/lib/hpux64:/opt/packages/libxslt-1.1.26/inst/lib/hpux64:/opt/packages/libxml2-2.7.8/inst/lib/hpux64
> \
>
> --with-includes=/opt/packages/zlib-1.2.5/inst/include:/opt/packages/libxslt-1.1.26/inst/include:/opt/packages/libxml2-2.7.8/inst/include
> \
> --with-pgport=5678 CFLAGS=-mlp64 CC=gcc
> make
> while make check TEMP_CONFIG=$PWD/temp.conf NO_LOCALE=1; do :; done
> # When the loop stops, use the core file to make a stack trace. If it runs
> # for a week without stopping, give up.
>
Yes, that makes sense. I did what you suggested (except that I used git://
instead of https:// during cloning as the latter returns with an error;
I'll fix it later) and in the first run the core files got generated:
bash-4.1$ ls -l
./postgresql/src/test/regress/tmp_check/data/core.postgres.*
-rw------- 1 pgbfarm users 6672384 Oct 30 22:04
./postgresql/src/test/regress/tmp_check/data/core.postgres.1267
-rw------- 1 pgbfarm users 3088384 Oct 30 22:09
./postgresql/src/test/regress/tmp_check/data/core.postgres.5422
Here is the stack trace:
bash-4.1$ gdb ./postgresql/tmp_install/usr/local/pgsql/bin/postgres
> ./postgresql/src/test/regress/tmp_check/data/core.postgres.5422
>
> HP gdb 6.1 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
>
> Copyright 1986 - 2009 Free Software Foundation, Inc.
>
> Hewlett-Packard Wildebeest 6.1 (based on GDB) is covered by the
>
> GNU General Public License. Type "show copying" to see the conditions to
>
> change it and/or distribute copies. Type "show warranty" for
> warranty/support.
>
> ..
>
> Core was generated by `postgres'.
>
> Program terminated with signal 11, Segmentation fault.
>
> SEGV_MAPERR - Address not mapped to object
>
> #0 0x3fffffffff3fdbf0 in <unknown_procedure> ()
>
> (gdb) bt
>
> #0 0x3fffffffff3fdbf0 in <unknown_procedure> ()
>
> warning: Attempting to unwind past bad PC 0x3fffffffff3fdbf0
>
> #1 0x40000000003fdc00:0 in equalTupleDescs (tupdesc1=0x60000000001f65e0,
>
> tupdesc2=0x60000000001fba08)
>
> #2 0x40000000017f9660:0 in RelationClearRelation (
>
> relation=0x60000000001f3730, rebuild=true)
>
> #3 0x40000000017fa730:0 in RelationFlushRelation
> (relation=0x60000000001f3730)
>
> #4 0x40000000017fabb0:0 in RelationCacheInvalidateEntry
> (relationId=27272)
>
> #5 0x40000000017c8f20:0 in LocalExecuteInvalidationMessage (
>
> msg=0x60000000001a46b8)
>
> #6 0x40000000017c84e0:0 in ProcessInvalidationMessages (
>
> group=0x60000000001a43e4, func=0x87ffffffef7b7250)
>
> #7 0x40000000017cb420:0 in CommandEndInvalidationMessages ()
>
> #8 0x40000000006b1c50:0 in AtCCI_LocalCache ()
>
> #9 0x40000000006b0910:0 in CommandCounterIncrement ()
>
> #10 0x4000000000807130:0 in create_toast_table (rel=0x60000000001f3730,
>
> toastOid=0, toastIndexOid=0, reloptions=0, lockmode=8, check=true,
>
> OIDOldToast=0)
>
> #11 0x4000000000805ac0:0 in CheckAndCreateToastTable (relOid=27272,
>
> reloptions=0, lockmode=8, check=true, OIDOldToast=0) at toasting.c:88
>
> #12 0x4000000000805850:0 in AlterTableCreateToastTable (relOid=27272,
>
> reloptions=0, lockmode=8) at toasting.c:62
>
> #13 0x4000000000aa9a30:0 in ATRewriteCatalogs (wqueue=0x87ffffffffffc3b0,
>
> lockmode=8, context=0x87ffffffffffc590)
>
Since the results are not posted to the dashboard, I've attached the
script log, regression.diffs and temp.conf for reference.
--
Sandeep Thakkar
Attachments:
temp.conf (application/octet-stream, 154B) download
run.log (application/octet-stream, 905.5K) download
regression.diffs (application/octet-stream, 3.5M) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
@ 2021-10-31 19:01 ` Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 107+ messages in thread
From: Andrey Borodin @ 2021-10-31 19:01 UTC (permalink / raw)
To: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; +Cc: Noah Misch <noah@leadboat.com>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Semab Tariq <semab.tariq@enterprisedb.com>
> 31 окт. 2021 г., в 12:31, Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> написал(а):
>
> Here is the stack trace:
Thanks Sandeep!
So far I didn't come up with a clear understanding how this might happen.
The only idea I have:
1. It seems equalTupleDescs() got two valid pointers, probably with broken data.
2. Maybe relation->rd_rel (alloceted just before relation->rd_att) was used incorrectly?
3. This could happen if CLASS_TUPLE_SIZE is calculated wrong. Don't we need to MAXALIGN everything due to added sizeof(relminmxid)?
#define CLASS_TUPLE_SIZE \
(offsetof(FormData_pg_class,relminmxid) + sizeof(TransactionId))
The idea doesn't seem very strong but that's only 2 cents I have. Noah, what do you think?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
@ 2021-10-31 19:18 ` Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
1 sibling, 1 reply; 107+ messages in thread
From: Andres Freund @ 2021-10-31 19:18 UTC (permalink / raw)
To: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; +Cc: Noah Misch <noah@leadboat.com>; Andrey Borodin <x4mmm@yandex-team.ru>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Semab Tariq <semab.tariq@enterprisedb.com>
Hi,
On 2021-10-31 13:01:00 +0530, Sandeep Thakkar wrote:
> > #1 0x40000000003fdc00:0 in equalTupleDescs (tupdesc1=0x60000000001f65e0,
> >
> > tupdesc2=0x60000000001fba08)
Could you print out *tupdesc1, *tupdesc2? And best also
p tupdesc1->attrs[0]
p tupdesc1->attrs[1]
p tupdesc1->attrs[2]
p tupdesc2->attrs[0]
p tupdesc2->attrs[1]
p tupdesc2->attrs[2]
> > #2 0x40000000017f9660:0 in RelationClearRelation (
> >
> > relation=0x60000000001f3730, rebuild=true)
Hm, too bad that we don't have working line numbers for some reason?
> > #3 0x40000000017fa730:0 in RelationFlushRelation
> > (relation=0x60000000001f3730)
> >
> > #4 0x40000000017fabb0:0 in RelationCacheInvalidateEntry
> > (relationId=27272)
> >
> > #5 0x40000000017c8f20:0 in LocalExecuteInvalidationMessage (
> >
> > msg=0x60000000001a46b8)
> >
> > #6 0x40000000017c84e0:0 in ProcessInvalidationMessages (
> >
> > group=0x60000000001a43e4, func=0x87ffffffef7b7250)
> >
> > #7 0x40000000017cb420:0 in CommandEndInvalidationMessages ()
> >
> > #8 0x40000000006b1c50:0 in AtCCI_LocalCache ()
> >
> > #9 0x40000000006b0910:0 in CommandCounterIncrement ()
> >
> > #10 0x4000000000807130:0 in create_toast_table (rel=0x60000000001f3730,
> >
> > toastOid=0, toastIndexOid=0, reloptions=0, lockmode=8, check=true,
> >
> > OIDOldToast=0)
> >
> > #11 0x4000000000805ac0:0 in CheckAndCreateToastTable (relOid=27272,
> >
> > reloptions=0, lockmode=8, check=true, OIDOldToast=0) at toasting.c:88
> >
> > #12 0x4000000000805850:0 in AlterTableCreateToastTable (relOid=27272,
> >
> > reloptions=0, lockmode=8) at toasting.c:62
> >
> > #13 0x4000000000aa9a30:0 in ATRewriteCatalogs (wqueue=0x87ffffffffffc3b0,
> >
> > lockmode=8, context=0x87ffffffffffc590)
This crash could suggest that somehow the catalogs were corrupted and
that we're not reading back valid tupledescs from them. Hopefully we'll
know more after looking at the tupledescs.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
@ 2021-11-01 13:10 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Sandeep Thakkar @ 2021-11-01 13:10 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; Andrey Borodin <x4mmm@yandex-team.ru>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Semab Tariq <semab.tariq@enterprisedb.com>
On Mon, Nov 1, 2021 at 12:48 AM Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2021-10-31 13:01:00 +0530, Sandeep Thakkar wrote:
> > > #1 0x40000000003fdc00:0 in equalTupleDescs
> (tupdesc1=0x60000000001f65e0,
> > >
> > > tupdesc2=0x60000000001fba08)
>
> Could you print out *tupdesc1, *tupdesc2? And best also
> p tupdesc1->attrs[0]
> p tupdesc1->attrs[1]
> p tupdesc1->attrs[2]
> p tupdesc2->attrs[0]
> p tupdesc2->attrs[1]
> p tupdesc2->attrs[2]
>
> you mean make the changes in the .c files to print these values and rerun
the build? Can you please share the files where this needs to be done or
give me a patch?
>
> > > #2 0x40000000017f9660:0 in RelationClearRelation (
> > >
> > > relation=0x60000000001f3730, rebuild=true)
>
> Hm, too bad that we don't have working line numbers for some reason?
>
>
> > > #3 0x40000000017fa730:0 in RelationFlushRelation
> > > (relation=0x60000000001f3730)
> > >
> > > #4 0x40000000017fabb0:0 in RelationCacheInvalidateEntry
> > > (relationId=27272)
> > >
> > > #5 0x40000000017c8f20:0 in LocalExecuteInvalidationMessage (
> > >
> > > msg=0x60000000001a46b8)
> > >
> > > #6 0x40000000017c84e0:0 in ProcessInvalidationMessages (
> > >
> > > group=0x60000000001a43e4, func=0x87ffffffef7b7250)
> > >
> > > #7 0x40000000017cb420:0 in CommandEndInvalidationMessages ()
> > >
> > > #8 0x40000000006b1c50:0 in AtCCI_LocalCache ()
> > >
> > > #9 0x40000000006b0910:0 in CommandCounterIncrement ()
> > >
> > > #10 0x4000000000807130:0 in create_toast_table (rel=0x60000000001f3730,
> > >
> > > toastOid=0, toastIndexOid=0, reloptions=0, lockmode=8, check=true,
> > >
> > > OIDOldToast=0)
> > >
> > > #11 0x4000000000805ac0:0 in CheckAndCreateToastTable (relOid=27272,
> > >
> > > reloptions=0, lockmode=8, check=true, OIDOldToast=0) at
> toasting.c:88
> > >
> > > #12 0x4000000000805850:0 in AlterTableCreateToastTable (relOid=27272,
> > >
> > > reloptions=0, lockmode=8) at toasting.c:62
> > >
> > > #13 0x4000000000aa9a30:0 in ATRewriteCatalogs
> (wqueue=0x87ffffffffffc3b0,
> > >
> > > lockmode=8, context=0x87ffffffffffc590)
>
>
> This crash could suggest that somehow the catalogs were corrupted and
> that we're not reading back valid tupledescs from them. Hopefully we'll
> know more after looking at the tupledescs.
>
>
> Greetings,
>
> Andres Freund
>
--
Sandeep Thakkar
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
@ 2021-11-01 13:18 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-11-01 13:18 UTC (permalink / raw)
To: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; +Cc: Andres Freund <andres@anarazel.de>; Noah Misch <noah@leadboat.com>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Semab Tariq <semab.tariq@enterprisedb.com>
> 1 нояб. 2021 г., в 18:10, Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> написал(а):
>
>
>
> On Mon, Nov 1, 2021 at 12:48 AM Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2021-10-31 13:01:00 +0530, Sandeep Thakkar wrote:
> > > #1 0x40000000003fdc00:0 in equalTupleDescs (tupdesc1=0x60000000001f65e0,
> > >
> > > tupdesc2=0x60000000001fba08)
>
> Could you print out *tupdesc1, *tupdesc2? And best also
> p tupdesc1->attrs[0]
> p tupdesc1->attrs[1]
> p tupdesc1->attrs[2]
> p tupdesc2->attrs[0]
> p tupdesc2->attrs[1]
> p tupdesc2->attrs[2]
>
> you mean make the changes in the .c files to print these values and rerun the build? Can you please share the files where this needs to be done or give me a patch?
You can paste these commands into gdb. Just as you did with bt. It will print values.
Thank you!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-11-01 13:33 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 107+ messages in thread
From: Sandeep Thakkar @ 2021-11-01 13:33 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Andres Freund <andres@anarazel.de>; Noah Misch <noah@leadboat.com>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Semab Tariq <semab.tariq@enterprisedb.com>
On Mon, Nov 1, 2021 at 6:48 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
>
> > 1 нояб. 2021 г., в 18:10, Sandeep Thakkar <
> sandeep.thakkar@enterprisedb.com> написал(а):
> >
> >
> >
> > On Mon, Nov 1, 2021 at 12:48 AM Andres Freund <andres@anarazel.de>
> wrote:
> > Hi,
> >
> > On 2021-10-31 13:01:00 +0530, Sandeep Thakkar wrote:
> > > > #1 0x40000000003fdc00:0 in equalTupleDescs
> (tupdesc1=0x60000000001f65e0,
> > > >
> > > > tupdesc2=0x60000000001fba08)
> >
> > Could you print out *tupdesc1, *tupdesc2? And best also
> > p tupdesc1->attrs[0]
> > p tupdesc1->attrs[1]
> > p tupdesc1->attrs[2]
> > p tupdesc2->attrs[0]
> > p tupdesc2->attrs[1]
> > p tupdesc2->attrs[2]
> >
> > you mean make the changes in the .c files to print these values and
> rerun the build? Can you please share the files where this needs to be done
> or give me a patch?
>
> You can paste these commands into gdb. Just as you did with bt. It will
> print values.
>
> Here you go (and with full bt):
(gdb) bt
#0 0x3fffffffff3fdbf0 in <unknown_procedure> ()
#1 0x40000000003fdc00:0 in equalTupleDescs (tupdesc1=0x60000000001f65e0,
tupdesc2=0x60000000001fba08)
#2 0x40000000017f9660:0 in RelationClearRelation (
relation=0x60000000001f3730, rebuild=true)
#3 0x40000000017fa730:0 in RelationFlushRelation
(relation=0x60000000001f3730)
#4 0x40000000017fabb0:0 in RelationCacheInvalidateEntry (relationId=27272)
#5 0x40000000017c8f20:0 in LocalExecuteInvalidationMessage (
msg=0x60000000001a46b8)
#6 0x40000000017c84e0:0 in ProcessInvalidationMessages (
group=0x60000000001a43e4, func=0x87ffffffef7b7250)
#7 0x40000000017cb420:0 in CommandEndInvalidationMessages ()
#8 0x40000000006b1c50:0 in AtCCI_LocalCache ()
#9 0x40000000006b0910:0 in CommandCounterIncrement ()
#10 0x4000000000807130:0 in create_toast_table (rel=0x60000000001f3730,
toastOid=0, toastIndexOid=0, reloptions=0, lockmode=8, check=true,
OIDOldToast=0)
#11 0x4000000000805ac0:0 in CheckAndCreateToastTable (relOid=27272,
reloptions=0, lockmode=8, check=true, OIDOldToast=0) at toasting.c:88
#12 0x4000000000805850:0 in AlterTableCreateToastTable (relOid=27272,
reloptions=0, lockmode=8) at toasting.c:62
#13 0x4000000000aa9a30:0 in ATRewriteCatalogs (wqueue=0x87ffffffffffc3b0,
lockmode=8, context=0x87ffffffffffc590)
---Type <return> to continue, or q <return> to quit---
#14 0x4000000000aa6fc0:0 in ATController (parsetree=0x60000000000e04e0,
rel=0x60000000001f3730, cmds=0x60000000000e0488, recurse=true,
lockmode=8,
context=0x87ffffffffffc590)
#15 0x4000000000aa6410:0 in AlterTable (stmt=0x60000000000e04e0, lockmode=8,
context=0x87ffffffffffc590)
#16 0x40000000012f9a50:0 in ProcessUtilitySlow (pstate=0x6000000000117210,
pstmt=0x60000000000e0830,
queryString=0x60000000000df7f0 "ALTER TABLE attmp ADD COLUMN c text;",
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x60000000000e0920, qc=0x87ffffffffffd838)
#17 0x40000000012f8890:0 in standard_ProcessUtility
(pstmt=0x60000000000e0830,
queryString=0x60000000000df7f0 "ALTER TABLE attmp ADD COLUMN c text;",
readOnlyTree=false, context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
queryEnv=0x0, dest=0x60000000000e0920, qc=0x87ffffffffffd838)
#18 0x40000000012f6150:0 in ProcessUtility (pstmt=0x60000000000e0830,
queryString=0x60000000000df7f0 "ALTER TABLE attmp ADD COLUMN c text;",
readOnlyTree=false, context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
queryEnv=0x0, dest=0x60000000000e0920, qc=0x87ffffffffffd838)
#19 0x40000000012f24c0:0 in PortalRunUtility (portal=0x600000000015baf0,
pstmt=0x60000000000e0830, isTopLevel=true, setHoldSnapshot=false,
dest=0x60000000000e0920, qc=0x87ffffffffffd838)
#20 0x40000000012f2c90:0 in PortalRunMulti (portal=0x600000000015baf0,
isTopLevel=true, setHoldSnapshot=false, dest=0x60000000000e0920,
---Type <return> to continue, or q <return> to quit---
altdest=0x60000000000e0920, qc=0x87ffffffffffd838)
#21 0x40000000012f08d0:0 in PortalRun (portal=0x600000000015baf0,
count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0x60000000000e0920, altdest=0x60000000000e0920,
qc=0x87ffffffffffd838)
#22 0x40000000012dcf70:0 in exec_simple_query (
query_string=0x60000000000df7f0 "ALTER TABLE attmp ADD COLUMN c text;")
#23 0x40000000012eaaf0:0 in PostgresMain (
dbname=0x60000000000aaa78 "regression",
username=0x60000000000a8ff8 "pgbfarm")
#24 0x40000000010751b0:0 in BackendRun (port=0x60000000001168e0)
#25 0x4000000001074040:0 in BackendStartup (port=0x60000000001168e0)
#26 0x4000000001068e60:0 in ServerLoop ()
#27 0x40000000010679b0:0 in PostmasterMain (argc=8, argv=0x87ffffffffffe610)
#28 0x4000000000d45660:0 in main (argc=8, argv=0x87ffffffffffe610)
at main.c:146
(gdb) p tupdesc1->attrs[1]
No symbol "tupdesc1" in current context.
(gdb) p tupdesc1->attrs[2]
No symbol "tupdesc1" in current context.
(gdb) p tupdesc2->attrs[0]
No symbol "tupdesc2" in current context.
(gdb) p tupdesc2->attrs[1]
No symbol "tupdesc2" in current context.
> (gdb)
>
Thank you!
>
> Best regards, Andrey Borodin.
--
Sandeep Thakkar
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
@ 2021-11-01 14:17 ` Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Robert Haas @ 2021-11-01 14:17 UTC (permalink / raw)
To: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; Noah Misch <noah@leadboat.com>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Semab Tariq <semab.tariq@enterprisedb.com>
On Mon, Nov 1, 2021 at 9:33 AM Sandeep Thakkar
<sandeep.thakkar@enterprisedb.com> wrote:
> (gdb) p tupdesc1->attrs[1]
> No symbol "tupdesc1" in current context.
> (gdb) p tupdesc1->attrs[2]
> No symbol "tupdesc1" in current context.
> (gdb) p tupdesc2->attrs[0]
> No symbol "tupdesc2" in current context.
> (gdb) p tupdesc2->attrs[1]
> No symbol "tupdesc2" in current context.
> (gdb)
I think you need to select stack frame 1 before running these
commands. I believe just running "frame 1" before you run these print
commands should do the trick.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
@ 2021-11-02 00:50 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Sandeep Thakkar @ 2021-11-02 00:50 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; Noah Misch <noah@leadboat.com>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Semab Tariq <semab.tariq@enterprisedb.com>
On Mon, Nov 1, 2021 at 7:47 PM Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Nov 1, 2021 at 9:33 AM Sandeep Thakkar
> <sandeep.thakkar@enterprisedb.com> wrote:
> > (gdb) p tupdesc1->attrs[1]
> > No symbol "tupdesc1" in current context.
> > (gdb) p tupdesc1->attrs[2]
> > No symbol "tupdesc1" in current context.
> > (gdb) p tupdesc2->attrs[0]
> > No symbol "tupdesc2" in current context.
> > (gdb) p tupdesc2->attrs[1]
> > No symbol "tupdesc2" in current context.
> > (gdb)
>
> I think you need to select stack frame 1 before running these
> commands. I believe just running "frame 1" before you run these print
> commands should do the trick.
>
> Thanks Robert, that worked. Here is the output:
(gdb) frame 1
#1 0x40000000003fdc00:0 in equalTupleDescs (tupdesc1=0x60000000001f65e0,
tupdesc2=0x60000000001fba08)
(gdb) p tupdesc1->attrs[0]
$1 = {attrelid = 27272, attname = {
data = "initial", '\000' <repeats 56 times>}, atttypid = 23,
attstattarget = -1, attlen = 4, attnum = 1, attndims = 0, attcacheoff = 0,
atttypmod = -1, attbyval = true, attalign = 105 'i', attstorage = 112 'p',
attcompression = 0 '\000', attnotnull = false, atthasdef = false,
atthasmissing = false, attidentity = 0 '\000', attgenerated = 0 '\000',
attisdropped = false, attislocal = true, attinhcount = 0, attcollation =
0}
(gdb) p tupdesc1->attrs[1]
$2 = {attrelid = 27272, attname = {data = "a", '\000' <repeats 62 times>},
atttypid = 23, attstattarget = -1, attlen = 4, attnum = 2, attndims = 0,
attcacheoff = -1, atttypmod = -1, attbyval = true, attalign = 105 'i',
attstorage = 112 'p', attcompression = 0 '\000', attnotnull = false,
atthasdef = true, atthasmissing = true, attidentity = 0 '\000',
attgenerated = 0 '\000', attisdropped = false, attislocal = true,
attinhcount = 0, attcollation = 0}
(gdb) p tupdesc1->attrs[2]
$3 = {attrelid = 27272, attname = {data = "b", '\000' <repeats 62 times>},
atttypid = 19, attstattarget = -1, attlen = 64, attnum = 3, attndims = 0,
attcacheoff = -1, atttypmod = -1, attbyval = false, attalign = 99 'c',
attstorage = 112 'p', attcompression = 0 '\000', attnotnull = false,
atthasdef = false, atthasmissing = false, attidentity = 0 '\000',
attgenerated = 0 '\000', attisdropped = false, attislocal = true,
attinhcount = 0, attcollation = 950}
(gdb) p tupdesc2->attrs[0]
$4 = {attrelid = 27272, attname = {
data = "initial", '\000' <repeats 56 times>}, atttypid = 23,
attstattarget = -1, attlen = 4, attnum = 1, attndims = 0, attcacheoff = 0,
atttypmod = -1, attbyval = true, attalign = 105 'i', attstorage = 112 'p',
attcompression = 0 '\000', attnotnull = false, atthasdef = false,
atthasmissing = false, attidentity = 0 '\000', attgenerated = 0 '\000',
attisdropped = false, attislocal = true, attinhcount = 0, attcollation =
0}
(gdb) p tupdesc2->attrs[1]
$5 = {attrelid = 27272, attname = {data = "a", '\000' <repeats 62 times>},
atttypid = 23, attstattarget = -1, attlen = 4, attnum = 2, attndims = 0,
attcacheoff = -1, atttypmod = -1, attbyval = true, attalign = 105 'i',
attstorage = 112 'p', attcompression = 0 '\000', attnotnull = false,
atthasdef = true, atthasmissing = true, attidentity = 0 '\000',
attgenerated = 0 '\000', attisdropped = false, attislocal = true,
attinhcount = 0, attcollation = 0}
(gdb) p tupdesc2->attrs[2]
$6 = {attrelid = 27272, attname = {data = "b", '\000' <repeats 62 times>},
atttypid = 19, attstattarget = -1, attlen = 64, attnum = 3, attndims = 0,
attcacheoff = -1, atttypmod = -1, attbyval = false, attalign = 99 'c',
attstorage = 112 'p', attcompression = 0 '\000', attnotnull = false,
atthasdef = false, atthasmissing = false, attidentity = 0 '\000',
attgenerated = 0 '\000', attisdropped = false, attislocal = true,
attinhcount = 0, attcollation = 950}
> (gdb)
>
--
> Robert Haas
> EDB: http://www.enterprisedb.com
>
--
Sandeep Thakkar
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
@ 2021-11-03 04:42 ` Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-03 04:42 UTC (permalink / raw)
To: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Semab Tariq <semab.tariq@enterprisedb.com>
On Tue, Nov 02, 2021 at 06:20:42AM +0530, Sandeep Thakkar wrote:
> (gdb) frame 1
>
> #1 0x40000000003fdc00:0 in equalTupleDescs (tupdesc1=0x60000000001f65e0,
>
> tupdesc2=0x60000000001fba08)
>
...
> (gdb) p tupdesc2->attrs[2]
>
> $6 = {attrelid = 27272, attname = {data = "b", '\000' <repeats 62 times>},
> atttypid = 19, attstattarget = -1, attlen = 64, attnum = 3, attndims = 0,
> attcacheoff = -1, atttypmod = -1, attbyval = false, attalign = 99 'c',
> attstorage = 112 'p', attcompression = 0 '\000', attnotnull = false,
> atthasdef = false, atthasmissing = false, attidentity = 0 '\000',
> attgenerated = 0 '\000', attisdropped = false, attislocal = true,
> attinhcount = 0, attcollation = 950}
That looks healthy. Since gdb isn't giving line numbers, let's single-step
from the start of the function and see if that is informative. Please apply
the attached patch, which just adds a test file. Then run "make -C
src/test/subscription check PROVE_TESTS=t/080_step_equalTupleDescs.pl" and
attach
src/test/subscription/tmp_check/log/regress_log_080_step_equalTupleDescs in a
reply to this email.
On Mon, Nov 01, 2021 at 12:01:08AM +0500, Andrey Borodin wrote:
> So far I didn't come up with a clear understanding how this might happen.
Agreed.
> The only idea I have:
> 1. It seems equalTupleDescs() got two valid pointers, probably with broken data.
> 2. Maybe relation->rd_rel (alloceted just before relation->rd_att) was used incorrectly?
> 3. This could happen if CLASS_TUPLE_SIZE is calculated wrong. Don't we need to MAXALIGN everything due to added sizeof(relminmxid)?
> #define CLASS_TUPLE_SIZE \
> (offsetof(FormData_pg_class,relminmxid) + sizeof(TransactionId))
See the comment at overread_tuplestruct_pg_cast for the reason why I think
that can't cause an actual malfunction. Still, there's some possibility of
this being the explanation.
Attachments:
step-to-crash-v0.patch (text/plain, 2.3K) download
diff --git a/src/test/subscription/t/080_step_equalTupleDescs.pl b/src/test/subscription/t/080_step_equalTupleDescs.pl
new file mode 100644
index 0000000..efa6dbd
--- /dev/null
+++ b/src/test/subscription/t/080_step_equalTupleDescs.pl
@@ -0,0 +1,94 @@
+# run with: make -C src/test/subscription check PROVE_TESTS=t/080_step_equalTupleDescs.pl
+# email src/test/subscription/tmp_check/log/regress_log_080_step_equalTupleDescs
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 1;
+use IPC::Run;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+my $in = '';
+my $out = '';
+my $timer = IPC::Run::timeout(180);
+my $psql = $node->background_psql('postgres', \$in, \$out, $timer,
+ on_error_stop => 0);
+
+$in .= q{
+SELECT pg_backend_pid(), 'after_pid';
+};
+$psql->pump until $out =~ /^([0-9]*)\|after_pid/m;
+my $pid = $1;
+print "backend pid: $pid\n";
+
+my $gdb_cmd_file = $node->basedir . '/commands-gdb';
+PostgreSQL::Test::Utils::append_to_file($gdb_cmd_file, q{
+b equalTupleDescs
+ignore 1 9
+set logging file } . $node->basedir . q{/gdb_started.txt
+set logging on
+set logging off
+continue
+p *tupdesc1
+p *tupdesc2
+p tupdesc1->attrs[0]
+p tupdesc1->attrs[1]
+p tupdesc1->attrs[2]
+p tupdesc2->attrs[0]
+p tupdesc2->attrs[1]
+p tupdesc2->attrs[2]
+backtrace full
+set $loopvar = 25000
+while ($loopvar--)
+ stepi
+end
+quit
+});
+
+my $gdb = IPC::Run::start(
+ [qw(gdb --batch --nx --pid), $pid, '-x', $gdb_cmd_file],
+ $timer);
+sleep .1 while !-f $node->basedir.'/gdb_started.txt';
+
+# first 28 lines of src/test/regress/sql/alter_table.sql
+$in .= q{
+--
+-- ALTER_TABLE
+--
+
+-- Clean up in case a prior regression run failed
+SET client_min_messages TO 'warning';
+DROP ROLE IF EXISTS regress_alter_table_user1;
+RESET client_min_messages;
+
+CREATE USER regress_alter_table_user1;
+
+--
+-- add attribute
+--
+
+CREATE TABLE attmp (initial int4);
+
+COMMENT ON TABLE attmp_wrong IS 'table comment';
+COMMENT ON TABLE attmp IS 'table comment';
+COMMENT ON TABLE attmp IS NULL;
+
+ALTER TABLE attmp ADD COLUMN xmin integer; -- fails
+
+ALTER TABLE attmp ADD COLUMN a int4 default 3;
+
+ALTER TABLE attmp ADD COLUMN b name;
+
+ALTER TABLE attmp ADD COLUMN c text;
+};
+
+$in .= q{
+\q
+};
+$psql->finish;
+$gdb->finish;
+ok(1);
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-03 08:03 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 107+ messages in thread
From: Semab Tariq @ 2021-11-03 08:03 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
Hi Noah
Sandeep is on vacation. So I am looking into this
I am facing some issues while applying the patch
bash-4.1$ git apply step-to-crash-v0.patch
fatal: git apply: bad git-diff - expected /dev/null on line 4
bash-4.1$ patch -p1 < step-to-crash-v0.patch
Hmm... I can't seem to find a patch in there anywhere.
Then i decided to manually create that
src/test/subscription/t/080_step_equalTupleDescs.pl file and placed the
content there once file is created i run this command make -C
src/test/subscription check PROVE_TESTS=t/080_step_equalTupleDescs.pl but
it didn't generated any
src/test/subscription/tmp_check/log/regress_log_080_step_equalTupleDescs
file
Also, I get this at the end of the make command
TAP tests not enabled. Try configuring with --enable-tap-tests
After enabling tap tests and executing the configure command again I get
this error message
checking for fop... no
checking for dbtoepub... no
checking for perl module IPC::Run 0.79... no
checking for perl module Test::More 0.87... no
checking for perl module Time::HiRes 1.52... ok
configure: error: Additional Perl modules are required to run TAP tests
Seems like can't enable tap tests with current frameworks
On Wed, Nov 3, 2021 at 9:42 AM Noah Misch <noah@leadboat.com> wrote:
> On Tue, Nov 02, 2021 at 06:20:42AM +0530, Sandeep Thakkar wrote:
> > (gdb) frame 1
> >
> > #1 0x40000000003fdc00:0 in equalTupleDescs (tupdesc1=0x60000000001f65e0,
> >
> > tupdesc2=0x60000000001fba08)
> >
> ...
> > (gdb) p tupdesc2->attrs[2]
> >
> > $6 = {attrelid = 27272, attname = {data = "b", '\000' <repeats 62
> times>},
> > atttypid = 19, attstattarget = -1, attlen = 64, attnum = 3, attndims =
> 0,
> > attcacheoff = -1, atttypmod = -1, attbyval = false, attalign = 99 'c',
> > attstorage = 112 'p', attcompression = 0 '\000', attnotnull = false,
> > atthasdef = false, atthasmissing = false, attidentity = 0 '\000',
> > attgenerated = 0 '\000', attisdropped = false, attislocal = true,
> > attinhcount = 0, attcollation = 950}
>
> That looks healthy. Since gdb isn't giving line numbers, let's single-step
> from the start of the function and see if that is informative. Please
> apply
> the attached patch, which just adds a test file. Then run "make -C
> src/test/subscription check PROVE_TESTS=t/080_step_equalTupleDescs.pl" and
> attach
> src/test/subscription/tmp_check/log/regress_log_080_step_equalTupleDescs
> in a
> reply to this email.
>
> On Mon, Nov 01, 2021 at 12:01:08AM +0500, Andrey Borodin wrote:
> > So far I didn't come up with a clear understanding how this might happen.
>
> Agreed.
>
> > The only idea I have:
> > 1. It seems equalTupleDescs() got two valid pointers, probably with
> broken data.
> > 2. Maybe relation->rd_rel (alloceted just before relation->rd_att) was
> used incorrectly?
> > 3. This could happen if CLASS_TUPLE_SIZE is calculated wrong. Don't we
> need to MAXALIGN everything due to added sizeof(relminmxid)?
> > #define CLASS_TUPLE_SIZE \
> > (offsetof(FormData_pg_class,relminmxid) + sizeof(TransactionId))
>
> See the comment at overread_tuplestruct_pg_cast for the reason why I think
> that can't cause an actual malfunction. Still, there's some possibility of
> this being the explanation.
>
--
Thanks & Regards,
Semab
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
@ 2021-11-04 00:50 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Thomas Munro @ 2021-11-04 00:50 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; +Cc: Noah Misch <noah@leadboat.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Wed, Nov 3, 2021 at 9:04 PM Semab Tariq <semab.tariq@enterprisedb.com> wrote:
> checking for fop... no
> checking for dbtoepub... no
> checking for perl module IPC::Run 0.79... no
> checking for perl module Test::More 0.87... no
> checking for perl module Time::HiRes 1.52... ok
> configure: error: Additional Perl modules are required to run TAP tests
>
> Seems like can't enable tap tests with current frameworks
You could always install that locally with CPAN, with something like:
export PERL5LIB=~/perl5/lib/perl5
cpan -T IPC::Run
Hmm, speaking of missing package management, GCC 4.6.0 is long dead
and missing many bug fixes, and in fact the whole 4.6.x line is
finished.
I remembered another phase of weird segmentation faults that was never
really explained, on that system:
https://www.postgresql.org/message-id/flat/CA%2BhUKGLukanJE9W8C%2B0n8iRsZDpbuhcWOxBMjGaUO-RNHhBGXw%4...
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
@ 2021-11-05 01:35 ` Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-05 01:35 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; Thomas Munro <thomas.munro@gmail.com>; +Cc: Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Thu, Nov 04, 2021 at 01:50:16PM +1300, Thomas Munro wrote:
> On Wed, Nov 3, 2021 at 9:04 PM Semab Tariq <semab.tariq@enterprisedb.com> wrote:
> > checking for fop... no
> > checking for dbtoepub... no
> > checking for perl module IPC::Run 0.79... no
> > checking for perl module Test::More 0.87... no
> > checking for perl module Time::HiRes 1.52... ok
> > configure: error: Additional Perl modules are required to run TAP tests
> >
> > Seems like can't enable tap tests with current frameworks
>
> You could always install that locally with CPAN, with something like:
>
> export PERL5LIB=~/perl5/lib/perl5
> cpan -T IPC::Run
Here's something more self-contained:
mkdir $HOME/perl-for-tap
(cd $HOME/perl-for-tap &&
wget http://backpan.perl.org/modules/by-authors/id/M/MS/MSCHWERN/Test-Simple-0.87_01.tar.gz &&
gzip -d <Test-Simple-0.87_01.tar.gz | tar xf - &&
wget https://cpan.metacpan.org/authors/id/R/RS/RSOD/IPC-Run-0.79.tar.gz &&
gzip -d <IPC-Run-0.79.tar.gz | tar xf -)
export PERL5LIB=$HOME/perl-for-tap/Test-Simple-0.87_01/lib:$HOME/perl-for-tap/IPC-Run-0.79/lib
perl -MIPC::Run -MTest::More -e 'print $IPC::Run::VERSION, "\n", $Test::More::VERSION, "\n"'
Semab, if you run those commands and then rerun the "configure" that failed,
does that help?
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-05 06:02 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Semab Tariq @ 2021-11-05 06:02 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Fri, Nov 5, 2021 at 6:35 AM Noah Misch <noah@leadboat.com> wrote:
> On Thu, Nov 04, 2021 at 01:50:16PM +1300, Thomas Munro wrote:
> > On Wed, Nov 3, 2021 at 9:04 PM Semab Tariq <semab.tariq@enterprisedb.com>
> wrote:
> > > checking for fop... no
> > > checking for dbtoepub... no
> > > checking for perl module IPC::Run 0.79... no
> > > checking for perl module Test::More 0.87... no
> > > checking for perl module Time::HiRes 1.52... ok
> > > configure: error: Additional Perl modules are required to run TAP tests
> > >
> > > Seems like can't enable tap tests with current frameworks
> >
> > You could always install that locally with CPAN, with something like:
> >
> > export PERL5LIB=~/perl5/lib/perl5
> > cpan -T IPC::Run
>
> Here's something more self-contained:
>
> mkdir $HOME/perl-for-tap
> (cd $HOME/perl-for-tap &&
> wget
> http://backpan.perl.org/modules/by-authors/id/M/MS/MSCHWERN/Test-Simple-0.87_01.tar.gz
> &&
> gzip -d <Test-Simple-0.87_01.tar.gz | tar xf - &&
> wget https://cpan.metacpan.org/authors/id/R/RS/RSOD/IPC-Run-0.79.tar.gz
> &&
> gzip -d <IPC-Run-0.79.tar.gz | tar xf -)
> export
> PERL5LIB=$HOME/perl-for-tap/Test-Simple-0.87_01/lib:$HOME/perl-for-tap/IPC-Run-0.79/lib
> perl -MIPC::Run -MTest::More -e 'print $IPC::Run::VERSION, "\n",
> $Test::More::VERSION, "\n"'
>
> Yes configure works with these commands thanks
After configure when i run make -C sarc/test/subscription check
PROVE_TESTS=t/080_step_equalTupleDescs.pl it got stucked I am attaching
regress_log_080_step_equalTupleDescs file and log file of make command
--
Thanks & Regards,
Semab
Attachments:
regress_log_080_step_equalTupleDescs (application/octet-stream, 3.9K) download
make.log (application/octet-stream, 2.3K) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
@ 2021-11-05 09:03 ` Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-05 09:03 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Fri, Nov 05, 2021 at 11:02:16AM +0500, Semab Tariq wrote:
> After configure when i run make -C sarc/test/subscription check
> PROVE_TESTS=t/080_step_equalTupleDescs.pl it got stucked I am attaching
> regress_log_080_step_equalTupleDescs file and log file of make command
Thanks. Please replace the previous test case patch with this one, and run it
again. (On your platform, gdb needs an explicit path to the executable.)
Attachments:
step-to-crash-v0.1.patch (text/plain, 2.6K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
diff --git a/src/test/subscription/t/080_step_equalTupleDescs.pl b/src/test/subscription/t/080_step_equalTupleDescs.pl
new file mode 100644
index 0000000..3d9f98d
--- /dev/null
+++ b/src/test/subscription/t/080_step_equalTupleDescs.pl
@@ -0,0 +1,99 @@
+# run with: make -C src/test/subscription check PROVE_TESTS=t/080_step_equalTupleDescs.pl
+# email src/test/subscription/tmp_check/log/regress_log_080_step_equalTupleDescs
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 1;
+use IPC::Run;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+my $in = '';
+my $out = '';
+my $timer = IPC::Run::timeout(180);
+my $psql = $node->background_psql('postgres', \$in, \$out, $timer,
+ on_error_stop => 0);
+
+$in .= q{
+SELECT pg_backend_pid(), 'after_pid';
+};
+$psql->pump until $out =~ /^([0-9]*)\|after_pid/m;
+my $pid = $1;
+print "backend pid: $pid\n";
+
+my $gdb_cmd_file = $node->basedir . '/commands-gdb';
+PostgreSQL::Test::Utils::append_to_file($gdb_cmd_file, q{
+b equalTupleDescs
+ignore 1 9
+set logging file } . $node->basedir . q{/gdb_started.txt
+set logging on
+set logging off
+continue
+p *tupdesc1
+p *tupdesc2
+p tupdesc1->attrs[0]
+p tupdesc1->attrs[1]
+p tupdesc1->attrs[2]
+p tupdesc2->attrs[0]
+p tupdesc2->attrs[1]
+p tupdesc2->attrs[2]
+backtrace full
+set $loopvar = 25000
+while ($loopvar--)
+ stepi
+end
+quit
+});
+
+$ENV{PATH} =~ /^([^:]*):/;
+my $bindir = $1;
+die "could not guess bindir from PATH=$ENV{PATH}" unless $1;
+
+my $gdb_cmdline = [qw(gdb --batch --nx --pid), $pid, '-x', $gdb_cmd_file,
+ "$bindir/postgres"];
+print("# Running: " . join(" ", @{$gdb_cmdline}) . "\n");
+my $gdb = IPC::Run::start($gdb_cmdline, $timer);
+sleep .1 while !-f $node->basedir.'/gdb_started.txt';
+
+# first 28 lines of src/test/regress/sql/alter_table.sql
+$in .= q{
+--
+-- ALTER_TABLE
+--
+
+-- Clean up in case a prior regression run failed
+SET client_min_messages TO 'warning';
+DROP ROLE IF EXISTS regress_alter_table_user1;
+RESET client_min_messages;
+
+CREATE USER regress_alter_table_user1;
+
+--
+-- add attribute
+--
+
+CREATE TABLE attmp (initial int4);
+
+COMMENT ON TABLE attmp_wrong IS 'table comment';
+COMMENT ON TABLE attmp IS 'table comment';
+COMMENT ON TABLE attmp IS NULL;
+
+ALTER TABLE attmp ADD COLUMN xmin integer; -- fails
+
+ALTER TABLE attmp ADD COLUMN a int4 default 3;
+
+ALTER TABLE attmp ADD COLUMN b name;
+
+ALTER TABLE attmp ADD COLUMN c text;
+};
+
+$in .= q{
+\q
+};
+$psql->finish;
+$gdb->finish;
+ok(1);
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-05 10:21 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Semab Tariq @ 2021-11-05 10:21 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Fri, Nov 5, 2021 at 2:03 PM Noah Misch <noah@leadboat.com> wrote:
> On Fri, Nov 05, 2021 at 11:02:16AM +0500, Semab Tariq wrote:
> > After configure when i run make -C sarc/test/subscription check
> > PROVE_TESTS=t/080_step_equalTupleDescs.pl it got stucked I am attaching
> > regress_log_080_step_equalTupleDescs file and log file of make command
>
> Thanks. Please replace the previous test case patch with this one, and
> run it
> again.
Did the same with the updated patch and this time make command exit with an
error
Bailout called. Further testing stopped: command "pg_ctl -D
/home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/tmp_check/t_080_step_equalTupleDescs_main_data/pgdata
-m immediate stop" exited with value 1 make: *** [check] Error 1 make:
Leaving directory
`/home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription'
PFA make command log and regress_log_080_step_equalTupleDescs files
--
Thanks & Regards,
Semab
Attachments:
make-1.log (application/octet-stream, 2.7K) download
regress_log_080_step_equalTupleDescs (application/octet-stream, 4.9K) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
@ 2021-11-05 11:22 ` Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-05 11:22 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Fri, Nov 05, 2021 at 03:21:15PM +0500, Semab Tariq wrote:
> Breakpoint 1, 0x40000000003fcb50:0 in equalTupleDescs (
> tupdesc1=0x40010006f968, tupdesc2=0x87ffffffffffac50)
The addresses there are weird. tupdesc1 is neither a stack address nor a heap
address; it may be in a program text section. tupdesc2 is a stack address.
In the earlier stack trace from
https://postgr.es/m/CANFyU94Xa8a5+4sZ7PxOiDLq+yN89g6y-9nNk-eLEvX6YUXbXA@mail.gmail.com
both tupdesc1 and tupdesc2 were heap addresses.
> /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/tmp_check/t_080_step_equalTupleDescs_main_data/commands-gdb:8: Error in sourced command file:
> Error accessing memory address 0x40010006f968: Bad address.
Thanks. Please try the attached test version, which avoids exiting too early
like the last version did.
Attachments:
step-to-crash-v0.2.patch (text/plain, 2.7K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
diff --git a/src/test/subscription/t/080_step_equalTupleDescs.pl b/src/test/subscription/t/080_step_equalTupleDescs.pl
new file mode 100644
index 0000000..f55bd5b
--- /dev/null
+++ b/src/test/subscription/t/080_step_equalTupleDescs.pl
@@ -0,0 +1,111 @@
+# run with: make -C src/test/subscription check PROVE_TESTS=t/080_step_equalTupleDescs.pl
+# email src/test/subscription/tmp_check/log/regress_log_080_step_equalTupleDescs
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 1;
+use IPC::Run;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+my $in = '';
+my $out = '';
+my $timer = IPC::Run::timeout(180);
+my $psql = $node->background_psql('postgres', \$in, \$out, $timer,
+ on_error_stop => 0);
+
+$in .= q{
+SELECT pg_backend_pid(), 'after_pid';
+};
+$psql->pump until $out =~ /^([0-9]*)\|after_pid/m;
+my $pid = $1;
+print "backend pid: $pid\n";
+
+$ENV{PATH} =~ /^([^:]*):/;
+my $bindir = $1;
+die "could not guess bindir from PATH=$ENV{PATH}" unless $1;
+
+my $gdb_in = '';
+my $gdb_cmdline = [qw(gdb --nx --pid), $pid, "$bindir/postgres"];
+print("# Running: " . join(" ", @{$gdb_cmdline}) . "\n");
+my $gdb = IPC::Run::start($gdb_cmdline, '<', \$gdb_in, $timer);
+
+$gdb_in = q{
+b equalTupleDescs
+ignore 1 9
+set logging file } . $node->basedir . q{/gdb_started.txt
+set logging on
+set logging off
+};
+$gdb->pump until $gdb_in eq '';
+sleep .1 while !-f $node->basedir.'/gdb_started.txt';
+
+# first 28 lines of src/test/regress/sql/alter_table.sql
+$in .= q{
+--
+-- ALTER_TABLE
+--
+
+-- Clean up in case a prior regression run failed
+SET client_min_messages TO 'warning';
+DROP ROLE IF EXISTS regress_alter_table_user1;
+RESET client_min_messages;
+
+CREATE USER regress_alter_table_user1;
+
+--
+-- add attribute
+--
+
+CREATE TABLE attmp (initial int4);
+
+COMMENT ON TABLE attmp_wrong IS 'table comment';
+COMMENT ON TABLE attmp IS 'table comment';
+COMMENT ON TABLE attmp IS NULL;
+
+ALTER TABLE attmp ADD COLUMN xmin integer; -- fails
+
+ALTER TABLE attmp ADD COLUMN a int4 default 3;
+
+ALTER TABLE attmp ADD COLUMN b name;
+
+ALTER TABLE attmp ADD COLUMN c text;
+};
+
+$in .= q{
+\q
+};
+
+$gdb_in .= q{
+continue
+p *tupdesc1
+p *tupdesc2
+p tupdesc1->attrs[0]
+p tupdesc1->attrs[1]
+p tupdesc1->attrs[2]
+p tupdesc2->attrs[0]
+p tupdesc2->attrs[1]
+p tupdesc2->attrs[2]
+backtrace full
+set $loopvar = 25000
+while ($loopvar--)
+ stepi
+end
+maintenance info sections ALLOBJ
+quit
+};
+
+for (;;) {
+ $psql->pump_nb;
+ $gdb->pump_nb;
+ last if $in eq '' && $gdb_in eq '';
+ sleep .1;
+}
+
+$psql->finish;
+$gdb->finish;
+ok(1);
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-06 23:08 ` Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-06 23:08 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Fri, Nov 05, 2021 at 04:22:36AM -0700, Noah Misch wrote:
> On Fri, Nov 05, 2021 at 03:21:15PM +0500, Semab Tariq wrote:
> > Breakpoint 1, 0x40000000003fcb50:0 in equalTupleDescs (
> > tupdesc1=0x40010006f968, tupdesc2=0x87ffffffffffac50)
>
> The addresses there are weird. tupdesc1 is neither a stack address nor a heap
> address; it may be in a program text section. tupdesc2 is a stack address.
> In the earlier stack trace from
> https://postgr.es/m/CANFyU94Xa8a5+4sZ7PxOiDLq+yN89g6y-9nNk-eLEvX6YUXbXA@mail.gmail.com
> both tupdesc1 and tupdesc2 were heap addresses.
>
> > /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/tmp_check/t_080_step_equalTupleDescs_main_data/commands-gdb:8: Error in sourced command file:
> > Error accessing memory address 0x40010006f968: Bad address.
>
> Thanks. Please try the attached test version, which avoids exiting too early
> like the last version did.
If you haven't run that yet, please use this version instead. It collects
more data. The log will probably be too big to be proper for the mailing
list, so please compress it.
Attachments:
step-to-crash-v0.3.patch (text/plain, 3.6K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
diff --git a/src/test/subscription/t/080_step_equalTupleDescs.pl b/src/test/subscription/t/080_step_equalTupleDescs.pl
new file mode 100644
index 0000000..5125af4
--- /dev/null
+++ b/src/test/subscription/t/080_step_equalTupleDescs.pl
@@ -0,0 +1,156 @@
+# run with: make -C src/test/subscription check PROVE_TESTS=t/080_step_equalTupleDescs.pl
+# email src/test/subscription/tmp_check/log/regress_log_080_step_equalTupleDescs
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+use IPC::Run;
+use Time::HiRes qw(usleep);
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+my $gdb_start_witness = $node->basedir . '/gdb_started.txt';
+
+$ENV{PATH} =~ /^([^:]*):/;
+my $bindir = $1;
+die "could not guess bindir from PATH=$ENV{PATH}" unless $1;
+
+for (1 .. 10)
+{
+ poll_start($node);
+ $node->safe_psql('postgres', 'DROP TABLE IF EXISTS attmp');
+ unlink $gdb_start_witness;
+
+ my $in = '';
+ my $out = '';
+ my $timer = IPC::Run::timeout(180);
+ my $psql = $node->background_psql('postgres', \$in, \$out, $timer,
+ on_error_stop => 0);
+
+ $in .= q{
+SELECT pg_backend_pid(), 'after_pid';
+};
+ $psql->pump until $out =~ /^([0-9]*)\|after_pid/m;
+ my $pid = $1;
+ print "backend pid: $pid\n";
+
+ my $gdb_in = '';
+ my $gdb_cmdline = [ qw(gdb --nx --pid), $pid, "$bindir/postgres" ];
+ print("# Running: " . join(" ", @{$gdb_cmdline}) . "\n");
+ my $gdb = IPC::Run::start($gdb_cmdline, '<', \$gdb_in, $timer);
+
+ $gdb_in = q{
+b equalTupleDescs
+ignore 1 9
+set logging file } . $gdb_start_witness . q{
+set logging on
+set logging off
+};
+ $gdb->pump until $gdb_in eq '';
+ sleep .1 while !-f $gdb_start_witness;
+
+ # first 28 lines of src/test/regress/sql/alter_table.sql
+ $in .= q{
+--
+-- ALTER_TABLE
+--
+
+-- Clean up in case a prior regression run failed
+SET client_min_messages TO 'warning';
+DROP ROLE IF EXISTS regress_alter_table_user1;
+RESET client_min_messages;
+
+CREATE USER regress_alter_table_user1;
+
+--
+-- add attribute
+--
+
+CREATE TABLE attmp (initial int4);
+
+COMMENT ON TABLE attmp_wrong IS 'table comment';
+COMMENT ON TABLE attmp IS 'table comment';
+COMMENT ON TABLE attmp IS NULL;
+
+ALTER TABLE attmp ADD COLUMN xmin integer; -- fails
+
+ALTER TABLE attmp ADD COLUMN a int4 default 3;
+
+ALTER TABLE attmp ADD COLUMN b name;
+
+ALTER TABLE attmp ADD COLUMN c text;
+};
+
+ $in .= q{
+\q
+};
+
+ $gdb_in .= q{
+continue
+p *tupdesc1
+p *tupdesc2
+p tupdesc1->attrs[0]
+p tupdesc1->attrs[1]
+p tupdesc1->attrs[2]
+p tupdesc2->attrs[0]
+p tupdesc2->attrs[1]
+p tupdesc2->attrs[2]
+backtrace full
+info proc mappings
+set $loopvar = 40
+while ($loopvar--)
+ frame $loopvar
+ info frame
+end
+set $loopvar = 25000
+while ($loopvar--)
+ stepi
+end
+maintenance info sections ALLOBJ
+info proc mappings
+quit
+};
+
+ for (;;)
+ {
+ $psql->pump_nb;
+ $gdb->pump_nb;
+ last if $in eq '' && $gdb_in eq '';
+ sleep .1;
+ }
+
+ $psql->finish;
+ $gdb->finish;
+ ok(1);
+
+ # Like $node->kill9, but don't care if a crash already downed the node.
+ kill(9, $node->{_pid});
+ $node->{_pid} = undef;
+}
+
+
+# copied from 017_shm.pl
+sub poll_start
+{
+ my ($node) = @_;
+
+ my $max_attempts = 180 * 10;
+ my $attempts = 0;
+
+ while ($attempts < $max_attempts)
+ {
+ $node->start(fail_ok => 1) && return 1;
+
+ # Wait 0.1 second before retrying.
+ usleep(100_000);
+
+ $attempts++;
+ }
+
+ # No success within 180 seconds. Try one last time without fail_ok, which
+ # will BAIL_OUT unless it succeeds.
+ $node->start && return 1;
+ return 0;
+}
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-07 15:25 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Semab Tariq @ 2021-11-07 15:25 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Sun, Nov 7, 2021 at 4:08 AM Noah Misch <noah@leadboat.com> wrote:
> On Fri, Nov 05, 2021 at 04:22:36AM -0700, Noah Misch wrote:
> > On Fri, Nov 05, 2021 at 03:21:15PM +0500, Semab Tariq wrote:
> > > Breakpoint 1, 0x40000000003fcb50:0 in equalTupleDescs (
> > > tupdesc1=0x40010006f968, tupdesc2=0x87ffffffffffac50)
> >
> > The addresses there are weird. tupdesc1 is neither a stack address nor
> a heap
> > address; it may be in a program text section. tupdesc2 is a stack
> address.
> > In the earlier stack trace from
> >
> https://postgr.es/m/CANFyU94Xa8a5+4sZ7PxOiDLq+yN89g6y-9nNk-eLEvX6YUXbXA@mail.gmail.com
> > both tupdesc1 and tupdesc2 were heap addresses.
> >
> > >
> /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/tmp_check/t_080_step_equalTupleDescs_main_data/commands-gdb:8:
> Error in sourced command file:
> > > Error accessing memory address 0x40010006f968: Bad address.
> >
> > Thanks. Please try the attached test version, which avoids exiting too
> early
> > like the last version did.
>
> If you haven't run that yet, please use this version instead. It collects
> more data. The log will probably be too big to be proper for the mailing
> list, so please compress it.
>
Hi Noah
PFA new regress_log_080_step_equalTupleDescs file generated from your
latest patch
--
Thanks & Regards,
Semab
Attachments:
log.tar.bz2 (application/x-bzip2, 49.0K) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
@ 2021-11-07 23:22 ` Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-07 23:22 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Sun, Nov 07, 2021 at 08:25:09PM +0500, Semab Tariq wrote:
> On Sun, Nov 7, 2021 at 4:08 AM Noah Misch <noah@leadboat.com> wrote:
> > On Fri, Nov 05, 2021 at 04:22:36AM -0700, Noah Misch wrote:
> > > On Fri, Nov 05, 2021 at 03:21:15PM +0500, Semab Tariq wrote:
> > > > Breakpoint 1, 0x40000000003fcb50:0 in equalTupleDescs (
> > > > tupdesc1=0x40010006f968, tupdesc2=0x87ffffffffffac50)
> > >
> > > The addresses there are weird. tupdesc1 is neither a stack address nor a heap
> > > address; it may be in a program text section. tupdesc2 is a stack address.
> > > In the earlier stack trace from
> > > https://postgr.es/m/CANFyU94Xa8a5+4sZ7PxOiDLq+yN89g6y-9nNk-eLEvX6YUXbXA@mail.gmail.com
> > > both tupdesc1 and tupdesc2 were heap addresses.
That turned out to be a false alarm. On gharial, a breakpoint at the start of
the function doesn't see the real arguments. After a ten-instruction
prologue, the real arguments appear, and they are heap addresses.
> PFA new regress_log_080_step_equalTupleDescs file generated from your latest patch
Thanks. That shows the crash happened sometime after strcmp(defval1->adbin,
defval2->adbin). Please run the attached version, which collects yet more
information.
Attachments:
step-to-crash-v0.4.patch (text/plain, 5.2K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
diff --git a/src/test/subscription/t/080_step_equalTupleDescs.pl b/src/test/subscription/t/080_step_equalTupleDescs.pl
new file mode 100644
index 0000000..a4dec82
--- /dev/null
+++ b/src/test/subscription/t/080_step_equalTupleDescs.pl
@@ -0,0 +1,223 @@
+# run with: make -C src/test/subscription check PROVE_TESTS=t/080_step_equalTupleDescs.pl
+# email src/test/subscription/tmp_check/log/regress_log_080_step_equalTupleDescs
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+use IPC::Run;
+use Time::HiRes qw(usleep);
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+my $gdb_start_witness = $node->basedir . '/gdb_started.txt';
+
+$ENV{PATH} =~ /^([^:]*):/;
+my $bindir = $1;
+die "could not guess bindir from PATH=$ENV{PATH}" unless $1;
+
+# I seem to recall gdb vintages where different attach methods had different
+# ability to find source code. Try that.
+my @attach_methods = (
+ # --pid alone
+ sub {
+ my $pid = shift;
+ return (
+ ['--pid', $pid],
+ '');
+ },
+ # --pid w/ explicit symbol-file
+ sub {
+ my $pid = shift;
+ return (
+ ['--pid', $pid],
+ "symbol-file -readnow $bindir/postgres");
+ },
+ # "attach" alone
+ sub {
+ my $pid = shift;
+ return (
+ [],
+ "attach $pid");
+ },
+ # "attach" w/ explicit symbol-file
+ sub {
+ my $pid = shift;
+ return (
+ [],
+ "symbol-file -readnow $bindir/postgres\nattach $pid");
+ },
+);
+
+foreach my $method (@attach_methods)
+{
+ poll_start($node);
+ $node->safe_psql('postgres', 'DROP TABLE IF EXISTS attmp');
+ unlink $gdb_start_witness;
+
+ my $in = '';
+ my $out = '';
+ my $timer = IPC::Run::timeout(180);
+ my $psql = $node->background_psql('postgres', \$in, \$out, $timer,
+ on_error_stop => 0);
+
+ $in .= q{
+SELECT pg_backend_pid(), 'after_pid';
+};
+ $psql->pump until $out =~ /^([0-9]*)\|after_pid/m;
+ my $pid = $1;
+ print "backend pid: $pid\n";
+
+ my($attach_args, $attach_cmd) = $method->($pid);
+ my $gdb_in = '';
+ my $gdb_cmdline = [ qw(gdb --nx), @$attach_args, "$bindir/postgres" ];
+ print("# Running: " . join(" ", @{$gdb_cmdline}) . "\n");
+ my $gdb = IPC::Run::start($gdb_cmdline, '<', \$gdb_in, $timer);
+
+ print("# Using attach command: $attach_cmd\n");
+ $gdb_in = q{
+set width 0
+#
+} . $attach_cmd . q{
+#
+b equalTupleDescs
+ignore 1 9
+#
+set logging file } . $gdb_start_witness . q{
+set logging on
+set logging off
+#
+echo disassembly attempts, not sure which will work on HP gdb 6.1\n
+disassemble /m equalTupleDescs
+disassemble 0x40000000003fcb50,0x40000000003fdfff
+disassemble 0x40000000003fcb50 0x40000000003fdfff
+echo source listing attempts -\
+ try it, but gdb is seeing line numbers for main.c and toasting.c only\n
+list *0x40000000003fdc30
+list *0x40000000003fdc30:2
+};
+ $gdb->pump until $gdb_in eq '';
+ sleep .1 while !-f $gdb_start_witness;
+
+ # first 28 lines of src/test/regress/sql/alter_table.sql
+ $in .= q{
+--
+-- ALTER_TABLE
+--
+
+-- Clean up in case a prior regression run failed
+SET client_min_messages TO 'warning';
+DROP ROLE IF EXISTS regress_alter_table_user1;
+RESET client_min_messages;
+
+CREATE USER regress_alter_table_user1;
+
+--
+-- add attribute
+--
+
+CREATE TABLE attmp (initial int4);
+
+COMMENT ON TABLE attmp_wrong IS 'table comment';
+COMMENT ON TABLE attmp IS 'table comment';
+COMMENT ON TABLE attmp IS NULL;
+
+ALTER TABLE attmp ADD COLUMN xmin integer; -- fails
+
+ALTER TABLE attmp ADD COLUMN a int4 default 3;
+
+ALTER TABLE attmp ADD COLUMN b name;
+
+ALTER TABLE attmp ADD COLUMN c text;
+};
+
+ $in .= q{
+\q
+};
+
+ $gdb_in .= q{
+echo Reach breakpoint.\n
+continue
+#
+echo On gharial, args are properly setup after a 10-instr preamble. Advance\
+ 11 instructions (in case I counted wrong) before capturing data.\n
+set $loopvar = 11
+while ($loopvar--)
+ stepi
+ x/i $pc
+end
+#
+echo Print any interesting memory state.\n
+p *tupdesc1
+p *tupdesc2
+p tupdesc1->attrs[0]
+p tupdesc1->attrs[1]
+p tupdesc1->attrs[2]
+p tupdesc2->attrs[0]
+p tupdesc2->attrs[1]
+p tupdesc2->attrs[2]
+p *tupdesc1->constr
+p *tupdesc2->constr
+p tupdesc1->constr->defval[0]
+p tupdesc2->constr->defval[0]
+backtrace full
+info proc mappings
+set $loopvar = 25
+while ($loopvar--)
+ frame $loopvar
+ info frame
+end
+#
+echo Advance to SIGSEGV, or give up after 25000 instructions.\n
+set $loopvar = 25000
+while ($loopvar--)
+ stepi
+ x/i $pc
+end
+maintenance info sections ALLOBJ
+info proc mappings
+quit
+};
+
+ for (;;)
+ {
+ $psql->pump_nb;
+ $gdb->pump_nb;
+ last if $in eq '' && $gdb_in eq '';
+ sleep .1;
+ }
+
+ $psql->finish;
+ $gdb->finish;
+ ok(1);
+
+ # Like $node->kill9, but don't care if a crash already downed the node.
+ kill(9, $node->{_pid});
+ $node->{_pid} = undef;
+}
+
+
+# copied from 017_shm.pl
+sub poll_start
+{
+ my ($node) = @_;
+
+ my $max_attempts = 180 * 10;
+ my $attempts = 0;
+
+ while ($attempts < $max_attempts)
+ {
+ $node->start(fail_ok => 1) && return 1;
+
+ # Wait 0.1 second before retrying.
+ usleep(100_000);
+
+ $attempts++;
+ }
+
+ # No success within 180 seconds. Try one last time without fail_ok, which
+ # will BAIL_OUT unless it succeeds.
+ $node->start && return 1;
+ return 0;
+}
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-08 06:22 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 07:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Semab Tariq @ 2021-11-08 06:22 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Mon, Nov 8, 2021 at 4:22 AM Noah Misch <noah@leadboat.com> wrote:
> On Sun, Nov 07, 2021 at 08:25:09PM +0500, Semab Tariq wrote:
> > On Sun, Nov 7, 2021 at 4:08 AM Noah Misch <noah@leadboat.com> wrote:
> > > On Fri, Nov 05, 2021 at 04:22:36AM -0700, Noah Misch wrote:
> > > > On Fri, Nov 05, 2021 at 03:21:15PM +0500, Semab Tariq wrote:
> > > > > Breakpoint 1, 0x40000000003fcb50:0 in equalTupleDescs (
> > > > > tupdesc1=0x40010006f968, tupdesc2=0x87ffffffffffac50)
> > > >
> > > > The addresses there are weird. tupdesc1 is neither a stack address
> nor a heap
> > > > address; it may be in a program text section. tupdesc2 is a stack
> address.
> > > > In the earlier stack trace from
> > > >
> https://postgr.es/m/CANFyU94Xa8a5+4sZ7PxOiDLq+yN89g6y-9nNk-eLEvX6YUXbXA@mail.gmail.com
> > > > both tupdesc1 and tupdesc2 were heap addresses.
>
> That turned out to be a false alarm. On gharial, a breakpoint at the
> start of
> the function doesn't see the real arguments. After a ten-instruction
> prologue, the real arguments appear, and they are heap addresses.
>
> > PFA new regress_log_080_step_equalTupleDescs file generated from your
> latest patch
>
> Thanks. That shows the crash happened sometime after
> strcmp(defval1->adbin,
> defval2->adbin). Please run the attached version,
PFA the new log file
--
Thanks & Regards,
Semab
Attachments:
log-1.tar.bz2 (application/x-bzip2, 70.4K) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
@ 2021-11-08 07:09 ` Noah Misch <noah@leadboat.com>
2021-11-08 09:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-08 07:09 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Mon, Nov 08, 2021 at 11:22:45AM +0500, Semab Tariq wrote:
> On Mon, Nov 8, 2021 at 4:22 AM Noah Misch <noah@leadboat.com> wrote:
> > Thanks. That shows the crash happened sometime after strcmp(defval1->adbin,
> > defval2->adbin). Please run the attached version,
>
> PFA the new log file
> 0x40000000003fdc30:2 in equalTupleDescs (tupdesc1=0x60000000001fdb98, tupdesc2=0x6000000000202da8)
> 0x40000000003fdc30:2 <equalTupleDescs+0x10e2>: br.call.sptk.many rp=0x3fffffffff3fdc30
> 0x3fffffffff3fdc30 in <unknown_procedure> ()
> 0x3fffffffff3fdc30: Error accessing memory address 0x3fffffffff3fdc30: Bad address.
This postgres binary apparently contains an explicit branch to
0x3fffffffff3fdc30, which is not an address reasonably expected to contain
code. (It's not a known heap, a known stack, or a CODE section from the
binary file.) This probably confirms a toolchain bug.
Would you do "git checkout 166f943" in the source directory you've been
testing, then rerun the test and post the compressed tmp_check/log directory?
I'm guessing that will show the bad branch instruction no longer present.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 07:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-08 09:09 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 14:46 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Semab Tariq @ 2021-11-08 09:09 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Mon, Nov 8, 2021 at 12:09 PM Noah Misch <noah@leadboat.com> wrote:
> On Mon, Nov 08, 2021 at 11:22:45AM +0500, Semab Tariq wrote:
> > On Mon, Nov 8, 2021 at 4:22 AM Noah Misch <noah@leadboat.com> wrote:
> > > Thanks. That shows the crash happened sometime after
> strcmp(defval1->adbin,
> > > defval2->adbin). Please run the attached version,
> >
> > PFA the new log file
>
> > 0x40000000003fdc30:2 in equalTupleDescs (tupdesc1=0x60000000001fdb98,
> tupdesc2=0x6000000000202da8)
> > 0x40000000003fdc30:2 <equalTupleDescs+0x10e2>:
> br.call.sptk.many rp=0x3fffffffff3fdc30
> > 0x3fffffffff3fdc30 in <unknown_procedure> ()
> > 0x3fffffffff3fdc30: Error accessing memory address 0x3fffffffff3fdc30:
> Bad address.
>
> This postgres binary apparently contains an explicit branch to
> 0x3fffffffff3fdc30, which is not an address reasonably expected to contain
> code. (It's not a known heap, a known stack, or a CODE section from the
> binary file.) This probably confirms a toolchain bug.
>
> Would you do "git checkout 166f943" in the source directory you've been
> testing, then rerun the test and post the compressed
> tmp_check/log directory?
>
So make command fails after I applied the v0.4 patch to 166f943 also it did
not create any tmp_check/log directory
PFA make output
I'm guessing that will show the bad branch instruction no longer present.
>
--
Thanks & Regards,
Semab
Attachments:
make-2.log (application/octet-stream, 4.0K) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 07:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 09:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
@ 2021-11-08 14:46 ` Noah Misch <noah@leadboat.com>
2021-11-09 06:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-08 14:46 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Mon, Nov 08, 2021 at 02:09:15PM +0500, Semab Tariq wrote:
> On Mon, Nov 8, 2021 at 12:09 PM Noah Misch <noah@leadboat.com> wrote:
> > This postgres binary apparently contains an explicit branch to
> > 0x3fffffffff3fdc30, which is not an address reasonably expected to contain
> > code. (It's not a known heap, a known stack, or a CODE section from the
> > binary file.) This probably confirms a toolchain bug.
> >
> > Would you do "git checkout 166f943" in the source directory you've been
> > testing, then rerun the test and post the compressed
> > tmp_check/log directory?
> >
>
> So make command fails after I applied the v0.4 patch to 166f943 also it did
> not create any tmp_check/log directory
> t/080_step_equalTupleDescs....Can't locate IPC/Run.pm in @INC (@INC contains: /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/../../../src/test/perl /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/../../../src/test/perl /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription /opt/perl_64/lib/5.8.8/IA64.ARCHREV_0-thread-multi-LP64 /opt/perl_64/lib/5.8.8 /opt/perl_64/lib/site_perl/5.8.8/IA64.ARCHREV_0-thread-multi-LP64 /opt/perl_64/lib/site_perl/5.8.8 /opt/perl_64/lib/site_perl /opt/perl_64/lib/vendor_perl/5.8.8/IA64.ARCHREV_0-thread-multi-LP64 /opt/perl_64/lib/vendor_perl/5.8.8 /opt/perl_64/lib/vendor_perl .) at /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/../../../src/test/perl/PostgreSQL/Test/Cluster.pm line 102.
It looks like this attempt did not use a shell with the environment variables
discussed upthread. Use these commands first to bring back the required
environment:
export LD_LIBRARY_PATH='/opt/uuid-1.6.2/inst/lib:/opt/packages/uuid-1.6.2/inst/lib:/opt/packages/krb5-1.11.3/inst/lib/hpux64:/opt/packages/libmemcached-0.46/inst/lib/hpux64:/opt/packages/libevent-2.0.10/inst/lib/hpux64:/opt/packages/expat-2.0.1/inst/lib/hpux64:/opt/packages/gdbm-1.8.3/inst/lib/hpux64:/opt/packages/openldap-2.4.24/inst/lib/hpux64:/opt/packages/proj-4.7.0/inst/lib:/opt/packages/geos-3.2.2/inst/lib:/opt/packages/db-5.1.19/inst/lib/hpux64:/opt/packages/freetype-2.4.4/inst/lib/hpux64:/opt/packages/tcltk-8.5.9/inst/lib/hpux64:/opt/packages/openssl-1.0.0d/inst/lib/hpux64:/opt/packages/editline-2.9/inst/lib/hpux64:/opt/packages/mutt-1.5.21/inst/lib/hpux64:/opt/packages/libidn-1.20/inst/lib/hpux64:/opt/packages/libxslt-1.1.26/inst/lib/hpux64:/opt/packages/libgcrypt-1.4.6/inst/lib/hpux64:/opt/packages/libgpg_error-1.10/inst/lib/hpux64:/opt/packages/libxml2-2.7.8/inst/lib/hpux64:/opt/packages/zlib-1.2.5/inst/lib/hpux64:/opt/packages/grep-2.7/inst/lib/hpux64:/opt/packages/pcre-8.12/inst/lib/hpux64:/opt/packages/ncurses-5.8/inst/lib/hpux64:/opt/packages/termcap-1.3.1/inst/lib/hpux64:/opt/packages/gettext-0.18.1.1/inst/lib/hpux64:/opt/packages/libiconv-1.13.1/inst/lib/hpux64:/opt/packages/sdk-10.2.0.5.0-hpux-ia64/instantclient_10_2/lib'
export PERL5LIB=$HOME/perl-for-tap/Test-Simple-0.87_01/lib:$HOME/perl-for-tap/IPC-Run-0.79/lib
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 07:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 09:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 14:46 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-09 06:55 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-09 14:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Semab Tariq @ 2021-11-09 06:55 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Mon, Nov 8, 2021 at 7:46 PM Noah Misch <noah@leadboat.com> wrote:
> On Mon, Nov 08, 2021 at 02:09:15PM +0500, Semab Tariq wrote:
> > On Mon, Nov 8, 2021 at 12:09 PM Noah Misch <noah@leadboat.com> wrote:
> > > This postgres binary apparently contains an explicit branch to
> > > 0x3fffffffff3fdc30, which is not an address reasonably expected to
> contain
> > > code. (It's not a known heap, a known stack, or a CODE section from
> the
> > > binary file.) This probably confirms a toolchain bug.
> > >
> > > Would you do "git checkout 166f943" in the source directory you've been
> > > testing, then rerun the test and post the compressed
> > > tmp_check/log directory?
> > >
> >
> > So make command fails after I applied the v0.4 patch to 166f943 also it
> did
> > not create any tmp_check/log directory
>
> > t/080_step_equalTupleDescs....Can't locate IPC/Run.pm in @INC (@INC
> contains:
> /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/../../../src/test/perl
> /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription
> /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/../../../src/test/perl
> /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription
> /opt/perl_64/lib/5.8.8/IA64.ARCHREV_0-thread-multi-LP64
> /opt/perl_64/lib/5.8.8
> /opt/perl_64/lib/site_perl/5.8.8/IA64.ARCHREV_0-thread-multi-LP64
> /opt/perl_64/lib/site_perl/5.8.8 /opt/perl_64/lib/site_perl
> /opt/perl_64/lib/vendor_perl/5.8.8/IA64.ARCHREV_0-thread-multi-LP64
> /opt/perl_64/lib/vendor_perl/5.8.8 /opt/perl_64/lib/vendor_perl .) at
> /home/pgbfarm/buildroot-gharial-HEAD/postgresql/src/test/subscription/../../../src/test/perl/PostgreSQL/Test/Cluster.pm
> line 102.
>
> It looks like this attempt did not use a shell with the environment
> variables
> discussed upthread.
>
Yes, I missed the trick PFA tmp_check.tar.bz2
--
Thanks & Regards,
Semab
Attachments:
tmp_check.tar.bz2 (application/x-bzip2, 358.6K) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 07:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 09:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 14:46 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-09 06:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
@ 2021-11-09 14:40 ` Noah Misch <noah@leadboat.com>
2021-11-09 21:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-11-09 14:40 UTC (permalink / raw)
To: Semab Tariq <semab.tariq@enterprisedb.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Tue, Nov 09, 2021 at 11:55:57AM +0500, Semab Tariq wrote:
> On Mon, Nov 8, 2021 at 7:46 PM Noah Misch <noah@leadboat.com> wrote:
> > On Mon, Nov 08, 2021 at 02:09:15PM +0500, Semab Tariq wrote:
> > > On Mon, Nov 8, 2021 at 12:09 PM Noah Misch <noah@leadboat.com> wrote:
> > > > This postgres binary apparently contains an explicit branch to
> > > > 0x3fffffffff3fdc30, which is not an address reasonably expected to contain
> > > > code. (It's not a known heap, a known stack, or a CODE section from the
> > > > binary file.) This probably confirms a toolchain bug.
> > > >
> > > > Would you do "git checkout 166f943" in the source directory you've been
> > > > testing, then rerun the test and post the compressed
> > > > tmp_check/log directory?
> PFA tmp_check.tar.bz2
Excellent. No crash, and the only difference in equalTupleDescs() code
generation is the branch destination addresses. At commit 70bef49, gharial's
toolchain generates the invalid branch destination:
$ diff -U0 <(cut -b47- disasm/70bef49/equalTupleDescs) <(cut -b47- disasm/166f943/equalTupleDescs)
inline patch 1: /dev/fd/62 (+5/-5 lines) download inline-1.patch
--- /dev/fd/63 2021-11-09 06:11:20.927444437 -0800
+++ /dev/fd/62 2021-11-09 06:11:20.926444428 -0800
@@ -100 +100 @@
- br.call.sptk.many rp=0x40000000003cdc20
+ br.call.sptk.many rp=0x40000000003cde20
@@ -658 +658 @@
- br.call.sptk.many rp=0x40000000003cdc20
+ br.call.sptk.many rp=0x40000000003cde20
@@ -817 +817 @@
- br.call.sptk.many rp=0x3fffffffff3fdc30
+ br.call.sptk.many rp=0x4000000000400c30
@@ -949 +949 @@
- br.call.sptk.many rp=0x40000000003cdc20
+ br.call.sptk.many rp=0x40000000003cde20
@@ -970 +970 @@
- br.call.sptk.many rp=0x40000000003cdc20
+ br.call.sptk.many rp=0x40000000003cde20
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 07:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 09:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 14:46 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-09 06:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-09 14:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-11-09 21:55 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-10 03:53 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Thomas Munro @ 2021-11-09 21:55 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Semab Tariq <semab.tariq@enterprisedb.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Wed, Nov 10, 2021 at 3:40 AM Noah Misch <noah@leadboat.com> wrote:
> $ diff -U0 <(cut -b47- disasm/70bef49/equalTupleDescs) <(cut -b47- disasm/166f943/equalTupleDescs)
> --- /dev/fd/63 2021-11-09 06:11:20.927444437 -0800
> +++ /dev/fd/62 2021-11-09 06:11:20.926444428 -0800
> @@ -100 +100 @@
> - br.call.sptk.many rp=0x40000000003cdc20
> + br.call.sptk.many rp=0x40000000003cde20
> @@ -658 +658 @@
> - br.call.sptk.many rp=0x40000000003cdc20
> + br.call.sptk.many rp=0x40000000003cde20
> @@ -817 +817 @@
> - br.call.sptk.many rp=0x3fffffffff3fdc30
> + br.call.sptk.many rp=0x4000000000400c30
> @@ -949 +949 @@
> - br.call.sptk.many rp=0x40000000003cdc20
> + br.call.sptk.many rp=0x40000000003cde20
> @@ -970 +970 @@
> - br.call.sptk.many rp=0x40000000003cdc20
> + br.call.sptk.many rp=0x40000000003cde20
>
> Since "git diff 70bef49 166f943" contains nothing that correlates with such a
> change, I'm concluding that this is a bug in gharial's toolchain.
Excellent detective work.
> It looks like gharial's automatic buildfarm runs have been paused for nine
> days. Feel free to unpause it. Also, I recommend using the buildfarm client
> setnotes.pl to add a note like 'Rare signal 11 from toolchain bug'. Months or
> years pass between these events. Here are all gharial "signal 11" failures,
> likely some of which have other causes:
Yeah I spent time investigating some of these at the time and I know
others did too.
IMHO we should hunt down dead toolchains and gently see if we can get
them updated. There is no point in testing PostgreSQL on every
commit on a compiler someone built from a tarball many years ago and
never updated with bug fixes.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-24 23:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 07:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-08 09:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 14:46 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-09 06:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-09 14:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-11-09 21:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Thomas Munro <thomas.munro@gmail.com>
@ 2021-11-10 03:53 ` Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 107+ messages in thread
From: Noah Misch @ 2021-11-10 03:53 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Semab Tariq <semab.tariq@enterprisedb.com>; Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>; Robert Haas <robertmhaas@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; CM Team <cm@enterprisedb.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Wed, Nov 10, 2021 at 10:55:13AM +1300, Thomas Munro wrote:
> On Wed, Nov 10, 2021 at 3:40 AM Noah Misch <noah@leadboat.com> wrote:
> > $ diff -U0 <(cut -b47- disasm/70bef49/equalTupleDescs) <(cut -b47- disasm/166f943/equalTupleDescs)
> > --- /dev/fd/63 2021-11-09 06:11:20.927444437 -0800
> > +++ /dev/fd/62 2021-11-09 06:11:20.926444428 -0800
> > @@ -100 +100 @@
> > - br.call.sptk.many rp=0x40000000003cdc20
> > + br.call.sptk.many rp=0x40000000003cde20
> > @@ -658 +658 @@
> > - br.call.sptk.many rp=0x40000000003cdc20
> > + br.call.sptk.many rp=0x40000000003cde20
> > @@ -817 +817 @@
> > - br.call.sptk.many rp=0x3fffffffff3fdc30
> > + br.call.sptk.many rp=0x4000000000400c30
> > @@ -949 +949 @@
> > - br.call.sptk.many rp=0x40000000003cdc20
> > + br.call.sptk.many rp=0x40000000003cde20
> > @@ -970 +970 @@
> > - br.call.sptk.many rp=0x40000000003cdc20
> > + br.call.sptk.many rp=0x40000000003cde20
> >
> > Since "git diff 70bef49 166f943" contains nothing that correlates with such a
> > change, I'm concluding that this is a bug in gharial's toolchain.
>
> Excellent detective work.
Thanks.
> > It looks like gharial's automatic buildfarm runs have been paused for nine
> > days. Feel free to unpause it. Also, I recommend using the buildfarm client
> > setnotes.pl to add a note like 'Rare signal 11 from toolchain bug'. Months or
> > years pass between these events. Here are all gharial "signal 11" failures,
> > likely some of which have other causes:
>
> Yeah I spent time investigating some of these at the time and I know
> others did too.
>
> IMHO we should hunt down dead toolchains and gently see if we can get
> them updated. There is no point in testing PostgreSQL on every
> commit on a compiler someone built from a tarball many years ago and
> never updated with bug fixes.
+1 for "gently see if". It's good that we have a few old-gcc animals, but
having versions of intermediate age is less important. Some owners will
decline, and that's okay.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-16 05:13 ` Noah Misch <noah@leadboat.com>
2021-08-22 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-08-16 05:13 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Sun, Aug 15, 2021 at 04:09:37PM +0500, Andrey Borodin wrote:
> > 15 авг. 2021 г., в 13:45, Noah Misch <noah@leadboat.com> написал(а):
> >> Do you see failures with that loop? If so, can you diagnose them?
> I do not observe failure on my laptop, though reproduced it on a linux server.
> I've fixed one bug in TwoPhaseGetXidByVXid(). Also rebased on actual master.
> > Just one 1PC failure in six hours of 1PC test runtime, though.
> I've attached a patch that reproduces the problem in 30sec on my server.
Having studied the broader inval situation, I found just one additional gap
that seemed potentially relevant. It didn't stop the failures under current
tests, however. The attached patch replaces my last patch on this thread, so
it should replace
v12-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcac.patch in your
series. (Like its predecessor, it's an unfinished proof-of-concept.)
With v12, on my machine, the same loop took 2000s to get three failures, both
of which were in the 1PC tests. I ran out of time to study the failure
mechanism. Would you diagnose what happens when it fails on your server?
Also see the larger review from Andres.
Attachments:
inval-build-race-v0.1.patch (text/plain, 2.3K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 13d9994..da6d8d6 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1228,10 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ inProgress[in_progress_offset].relid = InvalidOid;
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2805,6 +2824,13 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
@@ -2845,6 +2871,7 @@ RelationCacheInvalidate(void)
List *rebuildFirstList = NIL;
List *rebuildList = NIL;
ListCell *l;
+ int i;
/*
* Reload relation mapping data before starting to reconstruct cache.
@@ -2931,6 +2958,10 @@ RelationCacheInvalidate(void)
RelationClearRelation(relation, true);
}
list_free(rebuildList);
+
+ /* Any RelationBuildDesc() on the stack must start over. */
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ inProgress[i].invalidated = true;
}
/*
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-16 05:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-08-22 12:30 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-22 17:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-22 12:30 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
Andres, Noah, thanks for the review.
> 16 авг. 2021 г., в 10:13, Noah Misch <noah@leadboat.com> написал(а):
>
> With v12, on my machine, the same loop took 2000s to get three failures, both
> of which were in the 1PC tests. I ran out of time to study the failure
> mechanism. Would you diagnose what happens when it fails on your server?
I'm still in progress of investigation. With your fix for RelationBuildDesc() invals my reproduction is also quite slow - I get error within 10 minutes or so.
I've observed few times that same Xid was WaitXact()'es twice. Is it possible that taking ShareLock on running xid (taken from PGPGROC of vxid) is not a good way to wait for transaction completition?
Is it possible that xid is in PGPROC before backend acquires its own lock on xid?
> Also see the larger review from Andres.
Sure, I will address Andres's notes as long as patches will finally work without errors. I agree with the idea of not inventing dynamic list and other pointed problems.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-16 05:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-22 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-22 17:42 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-23 17:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-22 17:42 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
> 22 авг. 2021 г., в 17:30, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> I've observed few times that same Xid was WaitXact()'es twice. Is it possible that taking ShareLock on running xid (taken from PGPGROC of vxid) is not a good way to wait for transaction completition?
> Is it possible that xid is in PGPROC before backend acquires its own lock on xid?
Oh, that's it. We first publish xid in PGPROC and only then take a lock in lock manager on it.
Ok, I know how to fix this.
Currently when testing combination of all fixes I observe things like
'error running SQL: 'psql:<stdin>:1: ERROR: prepared transaction with identifier "a" is busy''
Looks like kind of race condition in tests.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-16 05:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-22 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-22 17:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-23 17:38 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-24 05:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2021-08-23 17:38 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
> 22 авг. 2021 г., в 22:42, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
>
> Currently when testing combination of all fixes I observe things like
> 'error running SQL: 'psql:<stdin>:1: ERROR: prepared transaction with identifier "a" is busy''
> Looks like kind of race condition in tests.
There was a race condition in deterministic 2PC test. Fixed with synchronisation points.
PFA patches that work on my machines.
I'm going to start fixing review notes if it will not break until tomorrow.
BTW are subtransaction anything special wrt CIC? is it worth to sprinkle some SAVEPOINTs here and there, just to be sure?
Thanks!
Best regards, Andrey Borodin.
Attachments: download all (6 files, .tar.gz)
v12-0001-Introduce-TAP-test-for-CIC.patch (application/octet-stream, 5.4K) download
From e5960aba551277c3efb52ca9fd40b87fe2d1ad64 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 26 Jul 2021 15:46:01 +0500
Subject: [PATCH v12 1/6] Introduce TAP test for CIC
Implement some pgbench probablistic testing to stress out CREATE INDEX CONCURRENTLY.
---
contrib/amcheck/t/002_cic.pl | 73 ++++++++++++++++++++
src/test/perl/PostgresNode.pm | 126 ++++++++++++++++++++++++++++++++++
2 files changed, 199 insertions(+)
create mode 100644 contrib/amcheck/t/002_cic.pl
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
new file mode 100644
index 0000000000..299f749214
--- /dev/null
+++ b/contrib/amcheck/t/002_cic.pl
@@ -0,0 +1,73 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 4;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+#
+# Stress CIC with pgbench
+#
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=200');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=200',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent transactions',
+ {
+ '002_pgbench_concurrent_transaction' =>
+ q(
+ BEGIN;
+ SELECT pg_sleep(0.001);
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8158ea5b2f..631eb5618e 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1912,6 +1912,132 @@ sub background_psql
=pod
+=item $node->pgbench($opts, $stat, $out, $err, $name, $files, @args)
+
+Invoke pgbench, with parameters and files.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $stat
+
+Expected exit status.
+
+=item $out
+
+Reference to a regexp list that must match stdout.
+
+=item $err
+
+Reference to a regexp list that must match stderr.
+
+=item $name
+
+Name of test for error messages.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=item @args
+
+Further raw options or arguments.
+
+=back
+
+=cut
+sub pgbench
+{
+ my ($self, $opts, $stat, $out, $err, $name, $files, @args) = @_;
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ push @cmd, @args;
+
+ $self->command_checks_all(\@cmd, $stat, $out, $err, $name);
+
+ # cleanup?
+ #unlink @filenames or die "cannot unlink files (@filenames): $!";
+
+ return;
+}
+=pod
+
+=item $node->background_pgbench($dbname, \$stdin, \$stdout, $timer, $files, %opts) => harness
+
+Invoke B<pgbench> in background alike background_psql.
+
+=cut
+sub background_pgbench
+{
+ my ($self, $dbname, $stdin, $stdout, $timer, $files, $opts) = @_;
+
+ my @cmd = ('pgbench', split /\s+/, $opts);
+ my @filenames = ();
+ if (defined $files)
+ {
+
+ # note: files are ordered for determinism
+ for my $fn (sort keys %$files)
+ {
+ my $filename = $self->basedir . '/' . $fn;
+ push @cmd, '-f', $filename;
+
+ # cleanup file weight
+ $filename =~ s/\@\d+$//;
+
+ #push @filenames, $filename;
+ # filenames are expected to be unique on a test
+ if (-e $filename)
+ {
+ ok(0, "$filename must not already exist");
+ unlink $filename or die "cannot unlink $filename: $!";
+ }
+ TestLib::append_to_file($filename, $$files{$fn});
+ }
+ }
+
+ local %ENV = $self->_get_env();
+
+ # Ensure there is no data waiting to be sent:
+ $$stdin = "" if ref($stdin);
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd,
+ '<', $stdin, '>', $stdout, $timer;
+
+ # Not checking actual input unlike in background_psql
+
+ return $harness;
+}
+=pod
+
=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
--
2.24.3 (Apple Git-128)
=
v12-0002-PoC-fix-for-race-in-RelationBuildDesc-and-relcac.patch (application/octet-stream, 2.2K) download
From bd0961461e6c84d2cbe8cf20b5f6834c2fba8e03 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Fri, 30 Jul 2021 14:40:16 +0500
Subject: [PATCH v12 2/6] PoC fix for race in RelationBuildDesc() and relcache
invalidation
---
src/backend/utils/cache/relcache.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 13d9994af3..7eec7b1f41 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -997,9 +997,16 @@ equalRSDesc(RowSecurityDesc *rsdesc1, RowSecurityDesc *rsdesc2)
* (suggesting we are trying to access a just-deleted relation).
* Any other error is reported via elog.
*/
+typedef struct InProgressRels {
+ Oid relid;
+ bool invalidated;
+} InProgressRels;
+static InProgressRels inProgress[100];
+
static Relation
RelationBuildDesc(Oid targetRelId, bool insertIt)
{
+ int in_progress_offset;
Relation relation;
Oid relid;
HeapTuple pg_class_tuple;
@@ -1033,6 +1040,14 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
}
#endif
+ for (in_progress_offset = 0;
+ OidIsValid(inProgress[in_progress_offset].relid);
+ in_progress_offset++)
+ ;
+ inProgress[in_progress_offset].relid = targetRelId;
+retry:
+ inProgress[in_progress_offset].invalidated = false;
+
/*
* find the tuple in pg_class corresponding to the given relation id
*/
@@ -1213,6 +1228,12 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
*/
heap_freetuple(pg_class_tuple);
+ if (inProgress[in_progress_offset].invalidated)
+ goto retry; /* TODO free old one */
+ /* inProgress is in fact the stack, we can safely remove it's top */
+ inProgress[in_progress_offset].relid = InvalidOid;
+ Assert(inProgress[in_progress_offset + 1].relid == InvalidOid);
+
/*
* Insert newly created relation into relcache hash table, if requested.
*
@@ -2805,6 +2826,13 @@ RelationCacheInvalidateEntry(Oid relationId)
relcacheInvalsReceived++;
RelationFlushRelation(relation);
}
+ else
+ {
+ int i;
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ if (inProgress[i].relid == relationId)
+ inProgress[i].invalidated = true;
+ }
}
/*
--
2.24.3 (Apple Git-128)
=
v12-0003-Introduce-TAP-test-for-2PC-with-CIC-behavior.patch (application/octet-stream, 4.4K) download
From afd732743b56fd816bfdc74b37ff1b4824e9b01c Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 8 Aug 2021 14:46:50 +0500
Subject: [PATCH v12 3/6] Introduce TAP test for 2PC with CIC behavior
Currently there are two tests:
1. Deterministic reproduction of the bug with 3 psql sessions
2. Probablistic reproduction of the bug with 2 pgbench sessions
---
contrib/amcheck/t/003_cic_2pc.pl | 148 +++++++++++++++++++++++++++++++
1 file changed, 148 insertions(+)
create mode 100644 contrib/amcheck/t/003_cic_2pc.pl
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
new file mode 100644
index 0000000000..7e86722b26
--- /dev/null
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -0,0 +1,148 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test for CREATE INDEX CONCURRENTLY with prepared transactions
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+
+use Test::More tests => 5;
+
+my ($node, $result);
+
+#
+# Test set-up
+#
+$node = PostgresNode->new('CIC_2PC_test');
+$node->init;
+$node->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+$node->append_conf('postgresql.conf', 'lock_timeout = 5000');
+$node->start;
+$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node->safe_psql('postgres', q(CREATE INDEX idx on tbl(i)));
+
+
+#
+# Test 1. Run 3 overlapping 2PC transactions with concurrent reindex rebuild
+#
+# We have two concurrent background psqls: main_h for transaction and
+# reindex_h for cuncurrent reindex. Also we commit transactions from independent
+# psql's sometimes.
+#
+
+my $main_in = '';
+my $main_out = '';
+my $main_timer = IPC::Run::timeout(180);
+
+my $main_h = $node->background_psql('postgres', \$main_in, \$main_out, $main_timer,
+ on_error_stop => 1);
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint1
+);
+pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
+
+my $reindex_in = '';
+my $reindex_out = '';
+my $reindex_timer = IPC::Run::timeout(180);
+my $reindex_h = $node->background_psql('postgres', \$reindex_in, \$reindex_out, $reindex_timer,
+ on_error_stop => 1);
+$reindex_in .= q(
+\echo start
+REINDEX TABLE CONCURRENTLY tbl;
+);
+pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+
+$main_in .= q(
+PREPARE TRANSACTION 'a';
+);
+
+$main_in .= q(
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint2
+);
+pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'b';
+BEGIN;
+INSERT INTO tbl VALUES(0);
+\echo syncpoint3
+);
+pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+
+$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
+
+$main_in .= q(
+PREPARE TRANSACTION 'c';
+COMMIT PREPARED 'c';
+);
+$main_h->pump_nb;
+
+$main_h->finish;
+$reindex_h->finish;
+
+$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
+is($result, '0', 'bt_index_check checks index');
+
+
+#
+# Test 2. Stress CIC+2PC with pgbench
+#
+
+# Fix broken index first
+$node->safe_psql('postgres', q(REINDEX TABLE CONCURRENTLY tbl;));
+
+# Run background pgbench with CIC. We cannot mix-in this script into single pgbench:
+# CIC will deadlock with itself occasionally.
+my $pgbench_in = '';
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_out, $pgbench_timer,
+ {
+ '002_pgbench_concurrent_cic' =>
+ q(
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ '--no-vacuum --client=1 --transactions=100');
+
+# Run pgbench.
+$node->pgbench(
+ '--no-vacuum --client=5 --transactions=100',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent CID and 2PC',
+ {
+ '002_pgbench_concurrent_2pc' =>
+ q(
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ PREPARE TRANSACTION 'c:client_id';
+ COMMIT PREPARED 'c:client_id';
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result = ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
+$node->safe_psql('postgres', q(DROP TABLE tbl;));
+$node->stop;
+done_testing();
+
+# AUX stuff, taken from 001_pgbench_with_server.pl
\ No newline at end of file
--
2.24.3 (Apple Git-128)
=
v12-0004-Fix-CREATE-INDEX-CONCURRENTLY-in-precence-of-vxi.patch (application/octet-stream, 7.8K) download
From 5139013ffc1d3d4fe92ca8e9ba98de843dabcfcc Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Mon, 19 Jul 2021 11:50:02 +0500
Subject: [PATCH v12 4/6] Fix CREATE INDEX CONCURRENTLY in precence of vxids
converted to 2pc
---
src/backend/access/transam/twophase.c | 47 +++++++++++-
src/backend/storage/lmgr/lock.c | 103 +++++++++++++++++++++-----
src/include/access/twophase.h | 14 ++++
3 files changed, 143 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 2156de187c..4823ded52b 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -459,14 +459,15 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
proc->pgprocno = gxact->pgprocno;
SHMQueueElemInit(&(proc->links));
proc->waitStatus = PROC_WAIT_STATUS_OK;
- /* We set up the gxact's VXID as InvalidBackendId/XID */
- proc->lxid = (LocalTransactionId) xid;
+ /* We set up the gxact's VXID as real for CIC purposes */
+ proc->lxid = MyProc->lxid;
proc->xid = xid;
Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
proc->statusFlags = 0;
proc->pid = 0;
- proc->backendId = InvalidBackendId;
+ /* May be backendId of startup process */
+ proc->backendId = MyBackendId;
proc->databaseId = databaseid;
proc->roleId = owner;
proc->tempNamespaceId = InvalidOid;
@@ -846,6 +847,46 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
return result;
}
+/*
+ * TwoPhaseGetXidByVXid
+ * Try to lookup for vxid among prepared xacts
+ */
+XidListEntry
+TwoPhaseGetXidByVXid(VirtualTransactionId vxid)
+{
+ int i;
+ XidListEntry result;
+ result.next = NULL;
+ result.xid = InvalidTransactionId;
+
+ LWLockAcquire(TwoPhaseStateLock, LW_SHARED);
+
+ for (i = 0; i < TwoPhaseState->numPrepXacts; i++)
+ {
+ GlobalTransaction gxact = TwoPhaseState->prepXacts[i];
+ PGPROC *proc = &ProcGlobal->allProcs[gxact->pgprocno];
+ VirtualTransactionId proc_vxid;
+ GET_VXID_FROM_PGPROC(proc_vxid, *proc);
+
+ if (VirtualTransactionIdEquals(vxid, proc_vxid))
+ {
+ if (result.xid != InvalidTransactionId)
+ {
+ /* Already has candidate - need to alloc some space */
+ XidListEntry *copy = palloc(sizeof(XidListEntry));
+ copy->next = result.next;
+ copy->xid = result.xid;
+ result.next = copy;
+ }
+ result.xid = gxact->xid;
+ }
+ }
+
+ LWLockRelease(TwoPhaseStateLock);
+
+ return result;
+}
+
/*
* TwoPhaseGetDummyBackendId
* Get the dummy backend ID for prepared transaction specified by XID
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..c56f413adb 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -387,6 +387,7 @@ static void LockRefindAndRelease(LockMethod lockMethodTable, PGPROC *proc,
bool decrement_strong_lock_count);
static void GetSingleProcBlockerStatusData(PGPROC *blocked_proc,
BlockedProcsData *data);
+static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait);
/*
@@ -4529,6 +4530,76 @@ VirtualXactLockTableCleanup(void)
}
}
+/*
+ * WaitXact
+ *
+ * Wait for xid completition if have xid. Otherwise try to find xid among
+ * two-phase procarray entries.
+ */
+static bool WaitXact(VirtualTransactionId vxid, TransactionId xid, bool wait)
+{
+ LockAcquireResult lar;
+ LOCKTAG tag;
+ XidListEntry xidlist;
+ XidListEntry *xidlist_ptr = NULL; /* pointer to TwoPhaseGetXidByVXid()s pallocs */
+ bool result;
+
+ if (TransactionIdIsValid(xid))
+ {
+ /* We know exact xid - no need to search in 2PC state */
+ xidlist.xid = xid;
+ xidlist.next = NULL;
+ }
+ else
+ {
+ /* You cant have vxid among 2PCs if you have no 2PCs */
+ if (max_prepared_xacts == 0)
+ return true;
+
+ /*
+ * We must search for vxids in 2pc state
+ * XXX: O(N*N) complexity where N is number of prepared xacts
+ */
+ xidlist = TwoPhaseGetXidByVXid(vxid);
+ /* Return if transaction is gone entirely */
+ if (!TransactionIdIsValid(xidlist.xid))
+ return true;
+ xidlist_ptr = xidlist.next;
+ }
+
+
+ while (true)
+ {
+ /* Iterate over possible xids */
+ Assert(TransactionIdIsValid(xidlist.xid));
+ SET_LOCKTAG_TRANSACTION(tag, xidlist.xid);
+ lar = LockAcquire(&tag, ShareLock, false, !wait);
+ if (lar == LOCKACQUIRE_NOT_AVAIL)
+ {
+ result = false;
+ break;
+ }
+
+ LockRelease(&tag, ShareLock, false);
+
+ if (xidlist.next == NULL)
+ {
+ result = true;
+ break;
+ }
+ xidlist = *xidlist.next;
+ }
+
+ /* Free all pallocs made by TwoPhaseGetXidByVXid() */
+ while (xidlist_ptr)
+ {
+ void *chunk = xidlist_ptr;
+ xidlist_ptr = xidlist_ptr->next;
+ pfree(chunk);
+ }
+ return result;
+}
+
/*
* VirtualXactLock
*
@@ -4541,25 +4612,18 @@ VirtualXactLockTableCleanup(void)
bool
VirtualXactLock(VirtualTransactionId vxid, bool wait)
{
- LOCKTAG tag;
- PGPROC *proc;
+ LOCKTAG tag;
+ PGPROC *proc;
+ TransactionId xid = InvalidTransactionId;
Assert(VirtualTransactionIdIsValid(vxid));
+ /*
+ * Already prepared transactions don't hold vxid locks. The
+ * LocalTransactionId is always a normal, locked XID.
+ */
if (VirtualTransactionIdIsPreparedXact(vxid))
- {
- LockAcquireResult lar;
-
- /*
- * Prepared transactions don't hold vxid locks. The
- * LocalTransactionId is always a normal, locked XID.
- */
- SET_LOCKTAG_TRANSACTION(tag, vxid.localTransactionId);
- lar = LockAcquire(&tag, ShareLock, false, !wait);
- if (lar != LOCKACQUIRE_NOT_AVAIL)
- LockRelease(&tag, ShareLock, false);
- return lar != LOCKACQUIRE_NOT_AVAIL;
- }
+ return WaitXact(vxid, vxid.localTransactionId, wait);
SET_LOCKTAG_VIRTUALTRANSACTION(tag, vxid);
@@ -4573,7 +4637,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
*/
proc = BackendIdGetProc(vxid.backendId);
if (proc == NULL)
- return true;
+ return WaitXact(vxid, InvalidTransactionId, wait);
/*
* We must acquire this lock before checking the backendId and lxid
@@ -4587,9 +4651,12 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
|| proc->fpLocalTransactionId != vxid.localTransactionId)
{
LWLockRelease(&proc->fpInfoLock);
- return true;
+ return WaitXact(vxid, InvalidTransactionId, wait);
}
+ /* Save the xid to test if transaction coverted to 2pc later */
+ xid = proc->xid;
+
/*
* If we aren't asked to wait, there's no need to set up a lock table
* entry. The transaction is still in progress, so just return false.
@@ -4641,7 +4708,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
(void) LockAcquire(&tag, ShareLock, false, false);
LockRelease(&tag, ShareLock, false);
- return true;
+ return WaitXact(vxid, xid, wait);
}
/*
diff --git a/src/include/access/twophase.h b/src/include/access/twophase.h
index e27e1a8fe8..a5f28d3a80 100644
--- a/src/include/access/twophase.h
+++ b/src/include/access/twophase.h
@@ -25,6 +25,17 @@
*/
typedef struct GlobalTransactionData *GlobalTransaction;
+/*
+ * XidListEntry is expected to be used as list very rarely. Under normal
+ * circumstances TwoPhaseGetXidByVXid() returns only one xid.
+ * But under certain conditions can return many xids or nothing.
+ */
+typedef struct XidListEntry
+{
+ TransactionId xid;
+ struct XidListEntry* next;
+} XidListEntry;
+
/* GUC variable */
extern PGDLLIMPORT int max_prepared_xacts;
@@ -58,6 +69,9 @@ extern void PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, RepOriginId origin_id);
extern void PrepareRedoRemove(TransactionId xid, bool giveWarning);
extern void restoreTwoPhaseData(void);
+
extern bool LookupGXact(const char *gid, XLogRecPtr prepare_at_lsn,
TimestampTz origin_prepare_timestamp);
+
+extern XidListEntry TwoPhaseGetXidByVXid(VirtualTransactionId vxid);
#endif /* TWOPHASE_H */
--
2.24.3 (Apple Git-128)
=
v12-0005-PoC-fix-clear-xid.patch (application/octet-stream, 1.8K) download
From 3292e928cc5f25eb63453c485d416792bcc62bb9 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sat, 7 Aug 2021 20:27:14 +0500
Subject: [PATCH v12 5/6] PoC fix clear xid
---
src/backend/access/transam/xact.c | 3 ++-
src/backend/utils/cache/relcache.c | 5 +++++
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 387f80419a..9b19d939eb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -2500,7 +2500,6 @@ PrepareTransaction(void)
* done *after* the prepared transaction has been marked valid, else
* someone may think it is unlocked and recyclable.
*/
- ProcArrayClearTransaction(MyProc);
/*
* In normal commit-processing, this is all non-critical post-transaction
@@ -2535,6 +2534,8 @@ PrepareTransaction(void)
PostPrepare_MultiXact(xid);
PostPrepare_Locks(xid);
+
+ ProcArrayClearTransaction(MyProc);
PostPrepare_PredicateLocks(xid);
ResourceOwnerRelease(TopTransactionResourceOwner,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7eec7b1f41..c72c4c255d 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -2873,6 +2873,7 @@ RelationCacheInvalidate(void)
List *rebuildFirstList = NIL;
List *rebuildList = NIL;
ListCell *l;
+ int i;
/*
* Reload relation mapping data before starting to reconstruct cache.
@@ -2959,6 +2960,10 @@ RelationCacheInvalidate(void)
RelationClearRelation(relation, true);
}
list_free(rebuildList);
+
+ /* Any RelationBuildDesc() on the stack must start over. */
+ for (i = 0; OidIsValid(inProgress[i].relid); i++)
+ inProgress[i].invalidated = true;
}
/*
--
2.24.3 (Apple Git-128)
=
v12-0006-Do-CIC-test-time-based-to-ensure-bug-repro.patch (application/octet-stream, 1.5K) download
From 4e8954268a6e351cde24dbef8ae43d65effaf9b4 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Sun, 15 Aug 2021 11:04:35 +0000
Subject: [PATCH v12 6/6] Do CIC test time-based to ensure bug repro
---
contrib/amcheck/t/002_cic.pl | 4 ++--
contrib/amcheck/t/003_cic_2pc.pl | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
index 299f749214..ca1644bb0d 100644
--- a/contrib/amcheck/t/002_cic.pl
+++ b/contrib/amcheck/t/002_cic.pl
@@ -41,11 +41,11 @@ my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_ou
SELECT bt_index_check('idx',true);
)
},
- '--no-vacuum --client=1 --transactions=200');
+ '--no-vacuum --client=1 --time=10');
# Run pgbench.
$node->pgbench(
- '--no-vacuum --client=5 --transactions=200',
+ '--no-vacuum --client=5 --time=10',
0,
[qr{actually processed}],
[qr{^$}],
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index 7e86722b26..021a47aa42 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -114,11 +114,11 @@ my $pgbench_h = $node->background_pgbench('postgres', \$pgbench_in, \$pgbench_ou
SELECT bt_index_check('idx',true);
)
},
- '--no-vacuum --client=1 --transactions=100');
+ '--no-vacuum --client=1 --time=10');
# Run pgbench.
$node->pgbench(
- '--no-vacuum --client=5 --transactions=100',
+ '--no-vacuum --client=5 --time=10',
0,
[qr{actually processed}],
[qr{^$}],
--
2.24.3 (Apple Git-128)
=
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-16 05:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-22 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-22 17:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-23 17:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-08-24 05:37 ` Noah Misch <noah@leadboat.com>
2021-08-24 06:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 107+ messages in thread
From: Noah Misch @ 2021-08-24 05:37 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>
On Mon, Aug 23, 2021 at 10:38:00PM +0500, Andrey Borodin wrote:
> > 22 авг. 2021 г., в 22:42, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
> > Currently when testing combination of all fixes I observe things like
> > 'error running SQL: 'psql:<stdin>:1: ERROR: prepared transaction with identifier "a" is busy''
> > Looks like kind of race condition in tests.
>
> There was a race condition in deterministic 2PC test. Fixed with synchronisation points.
> PFA patches that work on my machines.
> I'm going to start fixing review notes if it will not break until tomorrow.
That is great news.
> BTW are subtransaction anything special wrt CIC? is it worth to sprinkle some SAVEPOINTs here and there, just to be sure?
Not especially. The AssignTransactionId() "Ensure parent(s) have XIDs"
behavior may be important to CIC. In the test pgbench runs for scripts
002_pgbench_concurrent_2pc and 002_pgbench_concurrent_transaction, it likely
wouldn't hurt to pass two equal-weight test scripts, one of which uses
savepoints.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-27 12:50 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Peter Geoghegan <pg@bowt.ie>
2021-08-06 18:03 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-16 05:13 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
2021-08-22 12:30 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-22 17:42 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-23 17:38 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-24 05:37 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Noah Misch <noah@leadboat.com>
@ 2021-08-24 06:38 ` Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 107+ messages in thread
From: Peter Geoghegan @ 2021-08-24 06:38 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>
On Mon, Aug 23, 2021 at 10:37 PM Noah Misch <noah@leadboat.com> wrote:
> > There was a race condition in deterministic 2PC test. Fixed with synchronisation points.
> > PFA patches that work on my machines.
> > I'm going to start fixing review notes if it will not break until tomorrow.
>
> That is great news.
+1. Great detective work.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
@ 2020-12-23 07:23 ` Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-23 07:52 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 107+ messages in thread
From: Andrey Borodin @ 2020-12-23 07:23 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
> 21 дек. 2020 г., в 10:40, Michael Paquier <michael@paquier.xyz> написал(а):
>
> Hmm. The approach taken by the patch looks to be back-patchable.
I was checking that patch\test works in supported branches and everything seems to be fine down to REL_10_STABLE.
But my machines (Ubuntu18 and MacOS) report several fails in isolation tests on REL9_6_20\REL9_6_STABLE. Particularly eval-plan-qual-trigger, drop-index-concurrently-1, and async-notify. I do not observe problems with regular isolation tests though.
Do I understand correctly that check-world tests on buildfarm 'make check-prepared-txns' and the problem is somewhere in my machines? Or something is actually broken\outdated?
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-23 07:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
@ 2020-12-23 07:52 ` Michael Paquier <michael@paquier.xyz>
2020-12-25 09:05 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 107+ messages in thread
From: Michael Paquier @ 2020-12-23 07:52 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org
On Wed, Dec 23, 2020 at 12:23:28PM +0500, Andrey Borodin wrote:
> Do I understand correctly that check-world tests on buildfarm 'make
> check-prepared-txns' and the problem is somewhere in my machines? Or
> something is actually broken\outdated?
The buildfarm code has no trace of check-prepared-txns. And, FWIW, I
see no failures at the top of REL9_6_STABLE. Do you mean that this
happens only with your patch? Or do you mean that you see failures
using the stable branch of upstream? I have not tested the former,
but the latter works fine on my end.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 107+ messages in thread
* Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
2020-12-23 07:23 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-23 07:52 ` Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data Michael Paquier <michael@paquier.xyz>
@ 2020-12-25 09:05 ` Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 0 replies; 107+ messages in thread
From: Andrey Borodin @ 2020-12-25 09:05 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 23 дек. 2020 г., в 12:52, Michael Paquier <michael@paquier.xyz> написал(а):
>
> On Wed, Dec 23, 2020 at 12:23:28PM +0500, Andrey Borodin wrote:
>> Do I understand correctly that check-world tests on buildfarm 'make
>> check-prepared-txns' and the problem is somewhere in my machines? Or
>> something is actually broken\outdated?
>
> FWIW, I
> see no failures at the top of REL9_6_STABLE.
Thanks, Michael! The problem was indeed within my machines. maintainer-cleanup is not enough for make check-prepared-txns. Fresh real installation is necessary.
I've checked that test works down to REL9_5_STABLE with patch.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 107+ messages in thread
end of thread, other threads:[~2021-11-10 03:53 UTC | newest]
Thread overview: 107+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-12-19 17:13 CREATE INDEX CONCURRENTLY does not index prepared xact's data Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:22 ` Victor Yegorov <vyegorov@gmail.com>
2020-12-19 17:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:48 ` Victor Yegorov <vyegorov@gmail.com>
2020-12-19 18:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-20 18:10 ` Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-19 17:57 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-12-21 05:40 ` Michael Paquier <michael@paquier.xyz>
2020-12-21 07:24 ` Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-21 08:19 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-17 07:24 ` Noah Misch <noah@leadboat.com>
2021-01-22 05:44 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-22 05:49 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-01-24 02:27 ` Noah Misch <noah@leadboat.com>
2021-01-30 16:06 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-05-01 12:42 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 17:01 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-17 20:12 ` Noah Misch <noah@leadboat.com>
2021-07-18 05:17 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 00:30 ` Noah Misch <noah@leadboat.com>
2021-07-19 07:10 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-19 18:10 ` Noah Misch <noah@leadboat.com>
2021-07-19 18:41 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 19:38 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-20 21:49 ` Noah Misch <noah@leadboat.com>
2021-07-21 08:13 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-21 17:55 ` Noah Misch <noah@leadboat.com>
2021-07-23 16:33 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-23 22:30 ` Noah Misch <noah@leadboat.com>
2021-07-23 22:53 ` Peter Geoghegan <pg@bowt.ie>
2021-07-27 12:50 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 12:30 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-29 14:21 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 02:25 ` Noah Misch <noah@leadboat.com>
2021-07-30 10:42 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-30 18:41 ` Noah Misch <noah@leadboat.com>
2021-07-31 08:37 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-07-31 08:51 ` Peter Geoghegan <pg@bowt.ie>
2021-08-02 12:45 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-06 18:03 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 15:33 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 19:00 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-07 22:19 ` Noah Misch <noah@leadboat.com>
2021-08-08 11:31 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-08 16:37 ` Noah Misch <noah@leadboat.com>
2021-08-14 22:13 ` Noah Misch <noah@leadboat.com>
2021-08-15 08:45 ` Noah Misch <noah@leadboat.com>
2021-08-15 11:09 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-15 13:45 ` Andres Freund <andres@anarazel.de>
2021-08-29 07:27 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-29 18:09 ` Andres Freund <andres@anarazel.de>
2021-08-29 18:38 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-08 06:45 ` Noah Misch <noah@leadboat.com>
2021-09-20 04:41 ` Noah Misch <noah@leadboat.com>
2021-09-25 17:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-09-25 20:10 ` Noah Misch <noah@leadboat.com>
2021-10-17 15:12 ` Noah Misch <noah@leadboat.com>
2021-10-18 13:23 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-19 03:02 ` Noah Misch <noah@leadboat.com>
2021-10-24 05:00 ` Noah Misch <noah@leadboat.com>
2021-10-24 11:45 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:35 ` Noah Misch <noah@leadboat.com>
2021-10-28 14:04 ` Noah Misch <noah@leadboat.com>
2021-10-28 20:59 ` Thomas Munro <thomas.munro@gmail.com>
2021-10-29 02:49 ` Noah Misch <noah@leadboat.com>
2021-10-30 06:09 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 07:26 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 07:59 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 08:14 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-30 19:19 ` Noah Misch <noah@leadboat.com>
2021-10-31 07:31 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-10-31 19:01 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-31 19:18 ` Andres Freund <andres@anarazel.de>
2021-11-01 13:10 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 13:18 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-01 13:33 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-01 14:17 ` Robert Haas <robertmhaas@gmail.com>
2021-11-02 00:50 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2021-11-03 04:42 ` Noah Misch <noah@leadboat.com>
2021-11-03 08:03 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-04 00:50 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-05 01:35 ` Noah Misch <noah@leadboat.com>
2021-11-05 06:02 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 09:03 ` Noah Misch <noah@leadboat.com>
2021-11-05 10:21 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-05 11:22 ` Noah Misch <noah@leadboat.com>
2021-11-06 23:08 ` Noah Misch <noah@leadboat.com>
2021-11-07 15:25 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-07 23:22 ` Noah Misch <noah@leadboat.com>
2021-11-08 06:22 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 07:09 ` Noah Misch <noah@leadboat.com>
2021-11-08 09:09 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-08 14:46 ` Noah Misch <noah@leadboat.com>
2021-11-09 06:55 ` Semab Tariq <semab.tariq@enterprisedb.com>
2021-11-09 14:40 ` Noah Misch <noah@leadboat.com>
2021-11-09 21:55 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-10 03:53 ` Noah Misch <noah@leadboat.com>
2021-08-16 05:13 ` Noah Misch <noah@leadboat.com>
2021-08-22 12:30 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-22 17:42 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-23 17:38 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-08-24 05:37 ` Noah Misch <noah@leadboat.com>
2021-08-24 06:38 ` Peter Geoghegan <pg@bowt.ie>
2020-12-23 07:23 ` Andrey Borodin <x4mmm@yandex-team.ru>
2020-12-23 07:52 ` Michael Paquier <michael@paquier.xyz>
2020-12-25 09:05 ` Andrey Borodin <x4mmm@yandex-team.ru>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox