pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
18+ messages / 5 participants
[nested] [flat]
Commits from this thread:
2022-02-10 1b8462bf1d 2d96a66c2e 3cc89d9c38 eac3c40070 Fix back-patch of "Avoid race in RelationBuildDesc() ..."
* pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
@ 2021-10-24 01:40 Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 18+ messages in thread
From: Noah Misch @ 2021-10-24 01:40 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/fdd965d074d46765c295223b119ca437dbcac973
Modified Files
--------------
contrib/amcheck/t/002_cic.pl | 78 ++++++++++++++++
src/backend/utils/cache/inval.c | 12 ++-
src/backend/utils/cache/relcache.c | 115 +++++++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 118 +++++++----------------
src/include/utils/inval.h | 1 +
src/include/utils/relcache.h | 2 +-
src/test/perl/PostgresNode.pm | 134 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 368 insertions(+), 93 deletions(-)
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
@ 2022-02-08 21:13 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 18+ messages in thread
From: Tomas Vondra @ 2022-02-08 21:13 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; pgsql-committers@lists.postgresql.org
On 10/24/21 03:40, Noah Misch wrote:
> Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
>
> CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
> no later than each backend's next transaction start. That failed to
> hold when a backend absorbed a relevant invalidation in the middle of
> running RelationBuildDesc() on the CIC index. Queries that use the
> resulting index can silently fail to find rows. Fix this for future
> index builds by making RelationBuildDesc() loop until it finishes
> without accepting a relevant invalidation. It may be necessary to
> reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
> Back-patch to 9.6 (all supported versions).
>
> Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
> Freund.
>
> Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
>
Unfortunately, this seems to have broken CLOBBER_CACHE_ALWAYS builds.
Since this commit, initdb never completes due to infinite retrying over
and over (on the first RelationBuildDesc call).
We have a CLOBBER_CACHE_ALWAYS buildfarm machine "avocet", and that
currently looks like this (top):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2626 buildfa+ 20 0 202888 21416 20084 R 98.34 0.531 151507:16
/home/buildfarm/avocet/buildroot/REL9_6_STABLE/pgsql.build/tmp_install/home/buildfarm/avocet/buildroot/REL9_6_STABLE/inst/bin/postgres
--boot -x1 -F
Yep, that's 151507 minutes, i.e. 104 days in initdb :-/
I haven't looked at this very closely yet, but it seems the whole
problem is we do this at the very beginning:
in_progress_list[in_progress_offset].invalidated = false;
/*
* find the tuple in pg_class corresponding to the given relation id
*/
pg_class_tuple = ScanPgRelation(targetRelId, true, false);
which seems entirely self-defeating, because ScanPgRelation acquires a
lock (on pg_class), which accepts invalidations, which invalidates
system caches (in clobber_cache_always), which sets promptly sets
in_progress_list[in_progress_offset].invalidated = false;
guaranteeing an infinite loop.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2022-02-09 00:43 ` Andres Freund <andres@anarazel.de>
2022-02-09 01:23 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Michael Paquier <michael@paquier.xyz>
2022-02-09 01:25 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 02:04 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
0 siblings, 3 replies; 18+ messages in thread
From: Andres Freund @ 2022-02-09 00:43 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; pgsql-hackers@postgresql.org; pgsql-release@postgresql.org; +Cc: Noah Misch <noah@leadboat.com>; pgsql-committers@lists.postgresql.org
Hi,
On 2022-02-08 22:13:01 +0100, Tomas Vondra wrote:
> On 10/24/21 03:40, Noah Misch wrote:
> > Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
> >
> > CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
> > no later than each backend's next transaction start. That failed to
> > hold when a backend absorbed a relevant invalidation in the middle of
> > running RelationBuildDesc() on the CIC index. Queries that use the
> > resulting index can silently fail to find rows. Fix this for future
> > index builds by making RelationBuildDesc() loop until it finishes
> > without accepting a relevant invalidation. It may be necessary to
> > reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
> > Back-patch to 9.6 (all supported versions).
> >
> > Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
> > Freund.
> >
> > Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
> >
>
> Unfortunately, this seems to have broken CLOBBER_CACHE_ALWAYS builds. Since
> this commit, initdb never completes due to infinite retrying over and over
> (on the first RelationBuildDesc call).
Ugh. Do we need to do something about WRT the next set of minor releases? Is
there a a chance of this occuring in "real" workloads?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
@ 2022-02-09 01:23 ` Michael Paquier <michael@paquier.xyz>
2022-02-09 01:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 18+ messages in thread
From: Michael Paquier @ 2022-02-09 01:23 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>; pgsql-hackers@postgresql.org, pgsql-release@postgresql.org, Noah Misch <noah@leadboat.com>; pgsql-committers@lists.postgresql.org
On Tue, Feb 08, 2022 at 04:43:47PM -0800, Andres Freund wrote:
> Ugh. Do we need to do something about WRT the next set of minor
> releases?
The set of minor releases of this week has already been stamped, so
that's too late :/
> Is there a a chance of this occuring in "real" workloads?
Ugh++. The problem is that we would not really detect that
automatically, isn't it?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2022-02-09 01:23 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Michael Paquier <michael@paquier.xyz>
@ 2022-02-09 01:43 ` Andres Freund <andres@anarazel.de>
2022-02-09 02:24 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 18+ messages in thread
From: Andres Freund @ 2022-02-09 01:43 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>; pgsql-hackers@postgresql.org, pgsql-release@postgresql.org, Noah Misch <noah@leadboat.com>; pgsql-committers@lists.postgresql.org
Hi,
On 2022-02-09 10:23:06 +0900, Michael Paquier wrote:
> On Tue, Feb 08, 2022 at 04:43:47PM -0800, Andres Freund wrote:
> > Ugh. Do we need to do something about WRT the next set of minor
> > releases?
>
> The set of minor releases of this week has already been stamped, so
> that's too late :/
It's stamped, not tagged, so we could send out new tarballs. Or we could skip
a release number. IIRC we had to do something along those lines before.
> Ugh++. The problem is that we would not really detect that
> automatically, isn't it?
What do you mean with detect here?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2022-02-09 01:23 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Michael Paquier <michael@paquier.xyz>
2022-02-09 01:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
@ 2022-02-09 02:24 ` Michael Paquier <michael@paquier.xyz>
2022-02-09 05:44 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 18+ messages in thread
From: Michael Paquier @ 2022-02-09 02:24 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>; pgsql-hackers@postgresql.org, pgsql-release@postgresql.org, Noah Misch <noah@leadboat.com>; pgsql-committers@lists.postgresql.org
On Tue, Feb 08, 2022 at 05:43:34PM -0800, Andres Freund wrote:
> It's stamped, not tagged, so we could send out new tarballs. Or we could skip
> a release number. IIRC we had to do something along those lines before.
It does not matter now, but the release is stamped and tagged.
> What do you mean with detect here?
Well, we would not be able to see that something is stuck by default,
but Noah has just answered to my question by mentioning wait_timeout
in the buildfarm configuration.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2022-02-09 01:23 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Michael Paquier <michael@paquier.xyz>
2022-02-09 01:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2022-02-09 02:24 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Michael Paquier <michael@paquier.xyz>
@ 2022-02-09 05:44 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 18+ messages in thread
From: Tom Lane @ 2022-02-09 05:44 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Andres Freund <andres@anarazel.de>; Tomas Vondra <tomas.vondra@enterprisedb.com>; pgsql-hackers@postgresql.org, pgsql-release@postgresql.org, Noah Misch <noah@leadboat.com>; pgsql-committers@lists.postgresql.org
Michael Paquier <michael@paquier.xyz> writes:
> On Tue, Feb 08, 2022 at 05:43:34PM -0800, Andres Freund wrote:
>> It's stamped, not tagged, so we could send out new tarballs. Or we could skip
>> a release number. IIRC we had to do something along those lines before.
> It does not matter now, but the release is stamped and tagged.
Yeah, I see no need to do anything about this on an emergency
basis.
>> What do you mean with detect here?
> Well, we would not be able to see that something is stuck by default,
> but Noah has just answered to my question by mentioning wait_timeout
> in the buildfarm configuration.
The buildfarm's wait_timeout option isn't that helpful here, because
when it triggers, the client just goes belly-up *with no report*.
So even if the CCA animals had it on, you'd not notice unless you
started to wonder why they'd not reported lately.
I think that's a bug that ought to be fixed. I do agree that
wait_timeout ought to be finite by default, too.
regards, tom lane
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
@ 2022-02-09 01:25 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 01:53 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 18+ messages in thread
From: Tomas Vondra @ 2022-02-09 01:25 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org; pgsql-release@postgresql.org; +Cc: Noah Misch <noah@leadboat.com>; pgsql-committers@lists.postgresql.org
On 2/9/22 01:43, Andres Freund wrote:
> Hi,
>
> On 2022-02-08 22:13:01 +0100, Tomas Vondra wrote:
>> On 10/24/21 03:40, Noah Misch wrote:
>>> Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
>>>
>>> CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
>>> no later than each backend's next transaction start. That failed to
>>> hold when a backend absorbed a relevant invalidation in the middle of
>>> running RelationBuildDesc() on the CIC index. Queries that use the
>>> resulting index can silently fail to find rows. Fix this for future
>>> index builds by making RelationBuildDesc() loop until it finishes
>>> without accepting a relevant invalidation. It may be necessary to
>>> reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
>>> Back-patch to 9.6 (all supported versions).
>>>
>>> Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
>>> Freund.
>>>
>>> Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
>>>
>>
>> Unfortunately, this seems to have broken CLOBBER_CACHE_ALWAYS builds. Since
>> this commit, initdb never completes due to infinite retrying over and over
>> (on the first RelationBuildDesc call).
>
> Ugh. Do we need to do something about WRT the next set of minor releases? Is
> there a a chance of this occuring in "real" workloads?
>
AFAICS this only affects builds with CLOBBER_CACHE_ALWAYS, and anyone
running such build in production clearly likes painful things anyway.
But really, for the infinite loop to happen, building a relation
descriptor has to invalidate a cache. And I haven't found a way to do
that without the CLOBBER_CACHE_ALWAYS thing.
Also, all the November minor releases include this commit, and there
were no reports about this (pretty obvious) issue. Buildfarm did not
complain either (but an animal may be stuck for months and we would not
know about it).
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2022-02-09 01:25 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2022-02-09 01:53 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 18+ messages in thread
From: Andres Freund @ 2022-02-09 01:53 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: pgsql-hackers@postgresql.org, pgsql-release@postgresql.org, Noah Misch <noah@leadboat.com>; pgsql-committers@lists.postgresql.org
Hi,
On 2022-02-09 02:25:09 +0100, Tomas Vondra wrote:
> AFAICS this only affects builds with CLOBBER_CACHE_ALWAYS, and anyone
> running such build in production clearly likes painful things anyway.
Yea, realistically nobody does that.
> But really, for the infinite loop to happen, building a relation descriptor
> has to invalidate a cache. And I haven't found a way to do that without the
> CLOBBER_CACHE_ALWAYS thing.
Phew.
> Also, all the November minor releases include this commit, and there were no
> reports about this (pretty obvious) issue. Buildfarm did not complain either
> (but an animal may be stuck for months and we would not know about it).
Ah, somehow I thought that wasn't yet in the last set of releases. Phew #2.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
@ 2022-02-09 02:04 ` Noah Misch <noah@leadboat.com>
2022-02-09 05:41 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2 siblings, 1 reply; 18+ messages in thread
From: Noah Misch @ 2022-02-09 02:04 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>; pgsql-hackers@postgresql.org; pgsql-release@postgresql.org; pgsql-committers@lists.postgresql.org
On Tue, Feb 08, 2022 at 04:43:47PM -0800, Andres Freund wrote:
> On 2022-02-08 22:13:01 +0100, Tomas Vondra wrote:
> > On 10/24/21 03:40, Noah Misch wrote:
> > > Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
> > >
> > > CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
> > > no later than each backend's next transaction start. That failed to
> > > hold when a backend absorbed a relevant invalidation in the middle of
> > > running RelationBuildDesc() on the CIC index. Queries that use the
> > > resulting index can silently fail to find rows. Fix this for future
> > > index builds by making RelationBuildDesc() loop until it finishes
> > > without accepting a relevant invalidation. It may be necessary to
> > > reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
> > > Back-patch to 9.6 (all supported versions).
> > >
> > > Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
> > > Freund.
> > >
> > > Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
> > >
> >
> > Unfortunately, this seems to have broken CLOBBER_CACHE_ALWAYS builds. Since
> > this commit, initdb never completes due to infinite retrying over and over
> > (on the first RelationBuildDesc call).
Thanks for the report. I had added the debug_discard arguments of
InvalidateSystemCachesExtended() and RelationCacheInvalidate() to make the new
code survive a CREATE TABLE at debug_discard_caches=5. Apparently that's not
enough for initdb. I'll queue a task to look at it.
It's a good reminder to set wait_timeout on buildfarm animals. (I should take
that advice, too.)
> Ugh. Do we need to do something about WRT the next set of minor releases?
No, given that this code already debuted in the November releases.
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2022-02-09 02:04 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
@ 2022-02-09 05:41 ` Noah Misch <noah@leadboat.com>
2022-02-09 15:27 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 18+ messages in thread
From: Noah Misch @ 2022-02-09 05:41 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org; pgsql-committers@lists.postgresql.org
On Tue, Feb 08, 2022 at 06:04:03PM -0800, Noah Misch wrote:
> On Tue, Feb 08, 2022 at 04:43:47PM -0800, Andres Freund wrote:
> > On 2022-02-08 22:13:01 +0100, Tomas Vondra wrote:
> > > On 10/24/21 03:40, Noah Misch wrote:
> > > > Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
> > > >
> > > > CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
> > > > no later than each backend's next transaction start. That failed to
> > > > hold when a backend absorbed a relevant invalidation in the middle of
> > > > running RelationBuildDesc() on the CIC index. Queries that use the
> > > > resulting index can silently fail to find rows. Fix this for future
> > > > index builds by making RelationBuildDesc() loop until it finishes
> > > > without accepting a relevant invalidation. It may be necessary to
> > > > reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
> > > > Back-patch to 9.6 (all supported versions).
> > > >
> > > > Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
> > > > Freund.
> > > >
> > > > Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
> > > >
> > >
> > > Unfortunately, this seems to have broken CLOBBER_CACHE_ALWAYS builds. Since
> > > this commit, initdb never completes due to infinite retrying over and over
> > > (on the first RelationBuildDesc call).
>
> Thanks for the report. I had added the debug_discard arguments of
> InvalidateSystemCachesExtended() and RelationCacheInvalidate() to make the new
> code survive a CREATE TABLE at debug_discard_caches=5. Apparently that's not
> enough for initdb. I'll queue a task to look at it.
The explanation was more boring than that. v13 and earlier have an additional
InvalidateSystemCaches() call site, which I neglected to update. Here's the
fix I intend to push.
Attachments:
fix-inval-build-race-v1.patch (text/plain, 1.7K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
Fix back-patch of "Avoid race in RelationBuildDesc() ..."
The back-patch of commit fdd965d074d46765c295223b119ca437dbcac973 broke
CLOBBER_CACHE_ALWAYS for v9.6 through v13, because it updated the
InvalidateSystemCaches() call pertaining to CLOBBER_CACHE_RECURSIVELY
and not also the one pertaining to CLOBBER_CACHE_ALWAYS. Back-patch to
v13, v12, v11, and v10.
Reported by Tomas Vondra. Reviewed by FIXME.
Discussion: https://postgr.es/m/df7b4c0b-7d92-f03f-75c4-9e08b269a716@enterprisedb.com
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 8b0503f..8fe1f0d 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -715,27 +715,27 @@ AcceptInvalidationMessages(void)
* slows things by at least a factor of 10000, so I wouldn't suggest
* trying to run the entire regression tests that way. It's useful to try
* a few simple tests, to make sure that cache reload isn't subject to
* internal cache-flush hazards, but after you've done a few thousand
* recursive reloads it's unlikely you'll learn more.
*/
#if defined(CLOBBER_CACHE_ALWAYS)
{
static bool in_recursion = false;
if (!in_recursion)
{
in_recursion = true;
- InvalidateSystemCaches();
+ InvalidateSystemCachesExtended(true);
in_recursion = false;
}
}
#elif defined(CLOBBER_CACHE_RECURSIVELY)
{
static int recursion_depth = 0;
/* Maximum depth is arbitrary depending on your threshold of pain */
if (recursion_depth < 3)
{
recursion_depth++;
InvalidateSystemCachesExtended(true);
recursion_depth--;
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Andres Freund <andres@anarazel.de>
2022-02-09 02:04 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-09 05:41 ` Re: pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
@ 2022-02-09 15:27 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 0 replies; 18+ messages in thread
From: Tomas Vondra @ 2022-02-09 15:27 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org; pgsql-committers@lists.postgresql.org
On 2/9/22 06:41, Noah Misch wrote:
>
> The explanation was more boring than that. v13 and earlier have an additional
> InvalidateSystemCaches() call site, which I neglected to update. Here's the
> fix I intend to push.
I tried this patch on 10 and 13, and it seems to fix the issue. So +1.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 18+ messages in thread
* pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
@ 2021-10-24 01:40 Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 18+ messages in thread
From: Noah Misch @ 2021-10-24 01:40 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
Branch
------
REL_14_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/dde966efb286ddc222f17013a8cf4561a3b67a5e
Modified Files
--------------
contrib/amcheck/t/002_cic.pl | 78 ++++++++++++++++
src/backend/utils/cache/inval.c | 12 ++-
src/backend/utils/cache/relcache.c | 115 +++++++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 118 +++++++----------------
src/include/utils/inval.h | 1 +
src/include/utils/relcache.h | 2 +-
src/test/perl/PostgresNode.pm | 134 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 368 insertions(+), 93 deletions(-)
^ permalink raw reply [nested|flat] 18+ messages in thread
* pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
@ 2021-10-24 01:40 Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 18+ messages in thread
From: Noah Misch @ 2021-10-24 01:40 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
Branch
------
REL_13_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/2e33b43599ad3868ee180f149050444bf0a7ca15
Modified Files
--------------
contrib/amcheck/Makefile | 2 +
contrib/amcheck/t/002_cic.pl | 78 +++++++++++++++++++++
src/backend/utils/cache/inval.c | 12 +++-
src/backend/utils/cache/relcache.c | 115 +++++++++++++++++++++++++++++--
src/include/utils/inval.h | 1 +
src/include/utils/relcache.h | 2 +-
src/test/perl/PostgresNode.pm | 135 +++++++++++++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 336 insertions(+), 10 deletions(-)
^ permalink raw reply [nested|flat] 18+ messages in thread
* pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
@ 2021-10-24 01:40 Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 18+ messages in thread
From: Noah Misch @ 2021-10-24 01:40 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
Branch
------
REL_12_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/0869e53d3a077a6840dbcfe86d4b4fc6b2906fab
Modified Files
--------------
contrib/amcheck/Makefile | 2 +
contrib/amcheck/t/002_cic.pl | 78 +++++++++++++++++++++
src/backend/utils/cache/inval.c | 12 +++-
src/backend/utils/cache/relcache.c | 105 ++++++++++++++++++++++++++++-
src/include/utils/inval.h | 1 +
src/include/utils/relcache.h | 2 +-
src/test/perl/PostgresNode.pm | 135 +++++++++++++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 331 insertions(+), 5 deletions(-)
^ permalink raw reply [nested|flat] 18+ messages in thread
* pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
@ 2021-10-24 01:40 Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 18+ messages in thread
From: Noah Misch @ 2021-10-24 01:40 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
Branch
------
REL_11_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/5141e471b36add19fe9d6b461f9ea11c0ca6b373
Modified Files
--------------
src/backend/utils/cache/inval.c | 12 +++-
src/backend/utils/cache/relcache.c | 107 ++++++++++++++++++++++++++++-
src/bin/pgbench/t/022_cic.pl | 93 +++++++++++++++++++++++++
src/include/utils/inval.h | 1 +
src/include/utils/relcache.h | 2 +-
src/test/perl/PostgresNode.pm | 135 +++++++++++++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
7 files changed, 346 insertions(+), 5 deletions(-)
^ permalink raw reply [nested|flat] 18+ messages in thread
* pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
@ 2021-10-24 01:40 Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 18+ messages in thread
From: Noah Misch @ 2021-10-24 01:40 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
Branch
------
REL_10_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/db86746fd11e6d453e53b1550a46d325e3574aa7
Modified Files
--------------
src/backend/utils/cache/inval.c | 12 ++-
src/backend/utils/cache/relcache.c | 107 ++++++++++++++++++++++++-
src/bin/pgbench/t/022_cic.pl | 93 ++++++++++++++++++++++
src/include/utils/inval.h | 1 +
src/include/utils/relcache.h | 2 +-
src/test/perl/PostgresNode.pm | 155 +++++++++++++++++++++++++++++++++++++
src/test/perl/TestLib.pm | 43 ++++++++++
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 409 insertions(+), 5 deletions(-)
^ permalink raw reply [nested|flat] 18+ messages in thread
* pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR
@ 2021-10-24 01:40 Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 18+ messages in thread
From: Noah Misch @ 2021-10-24 01:40 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY.
CIC and REINDEX CONCURRENTLY assume backends see their catalog changes
no later than each backend's next transaction start. That failed to
hold when a backend absorbed a relevant invalidation in the middle of
running RelationBuildDesc() on the CIC index. Queries that use the
resulting index can silently fail to find rows. Fix this for future
index builds by making RelationBuildDesc() loop until it finishes
without accepting a relevant invalidation. It may be necessary to
reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices.
Back-patch to 9.6 (all supported versions).
Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres
Freund.
Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com
Branch
------
REL9_6_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/e428699cb3be0a7abbd5a83bfa5ec3e26c7fea84
Modified Files
--------------
src/backend/utils/cache/inval.c | 10 ++-
src/backend/utils/cache/relcache.c | 107 ++++++++++++++++++++++++-
src/bin/pgbench/t/022_cic.pl | 93 ++++++++++++++++++++++
src/include/utils/inval.h | 1 +
src/include/utils/relcache.h | 2 +-
src/test/perl/PostgresNode.pm | 155 +++++++++++++++++++++++++++++++++++++
src/test/perl/TestLib.pm | 43 ++++++++++
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 408 insertions(+), 4 deletions(-)
^ permalink raw reply [nested|flat] 18+ messages in thread
end of thread, other threads:[~2022-02-09 15:27 UTC | newest]
Thread overview: 18+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2022-02-08 21:13 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 00:43 ` Andres Freund <andres@anarazel.de>
2022-02-09 01:23 ` Michael Paquier <michael@paquier.xyz>
2022-02-09 01:43 ` Andres Freund <andres@anarazel.de>
2022-02-09 02:24 ` Michael Paquier <michael@paquier.xyz>
2022-02-09 05:44 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-02-09 01:25 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-09 01:53 ` Andres Freund <andres@anarazel.de>
2022-02-09 02:04 ` Noah Misch <noah@leadboat.com>
2022-02-09 05:41 ` Noah Misch <noah@leadboat.com>
2022-02-09 15:27 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
2021-10-24 01:40 pgsql: Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURR Noah Misch <noah@leadboat.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox