Date: Sun, 18 Jul 2021 20:30:09 -0400
From: Noah Misch <noah@leadboat.com>
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Michael Paquier <michael@paquier.xyz>, Tom Lane <tgl@sss.pgh.pa.us>,
	pgsql-bugs@lists.postgresql.org
Subject: Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data
Message-ID: <20210719003009.GA931573@gust.leadboat.com>
References: <FD04942E-BC4E-42FE-82DF-75D28E3093AC@yandex-team.ru>
 <2C1B82E7-B50C-470C-AE10-794ED2A7A64E@yandex-team.ru>
 <20210117072454.GA1709971@rfd.leadboat.com>
 <EF4A43BD-C62C-429B-94EE-9303A1321F47@yandex-team.ru>
 <20210124022743.GA2270535@rfd.leadboat.com>
 <74C78048-70E8-4356-8F10-6370C37136C8@yandex-team.ru>
 <9C782C9D-D53E-4154-B910-46F062A9AE9B@yandex-team.ru>
 <01824242-AA92-4FE9-9BA7-AEBAFFEA3D0C@yandex-team.ru>
 <20210717201240.GA874334@gust.leadboat.com>
 <A74C56AE-FA8A-4BBD-8941-06AA8FED73E1@yandex-team.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <A74C56AE-FA8A-4BBD-8941-06AA8FED73E1@yandex-team.ru>
Archived-At: 
 <https://www.postgresql.org/message-id/20210719003009.GA931573%40gust.leadboat.com>
Precedence: bulk

On Sun, Jul 18, 2021 at 10:17:28AM +0500, Andrey Borodin wrote:
> > 18 июля 2021 г., в 01:12, Noah Misch <noah@leadboat.com> написал(а):
> > 
> > Suppose some transaction has a vxid but no xid.  Shortly after
> > GetLockConflicts(), it acquires an xid, modifies the table, and issues PREPARE
> > TRANSACTION.  Could that cause a corrupt index even with this diff?
> 
> Firstly I've tried to stress things out. This little go program [0] easily reproduces corruption on patched code.

By "patched code", you mean return_xids_with_vxids.diff applied to recent
master, right?

> Meanwhile vxid->xid->2px program does not [1] (both patched and unpatched).
> 
> I think CIC does not care much about VXIDs at all. It's only important when real XID started: before GetLockConflicts() or after.

CIC likes VXIDs because they're assigned early in the xact, before any catalog
access.  Consider the DefineIndex() step "wait until no running transaction
could have the table open with the old list of indexes."  ExecOpenIndices()
can happen before AssignTransactionId(), so waiting on XIDs alone wouldn't
suffice.

To fix $SUBJECT, it sounds like we need a way to identify a transaction,
usable as early as the transaction's first catalog access and remaining valid
until COMMIT PREPARED finishes.  We may initially see a transaction as having
a VXID and no XID, then later need to wait for that transaction when it has
entered prepared state, having an XID and no VXID.  How might we achieve that?