MIME-Version: 1.0
References: 
 <CAH5HC94r2ZKy_MhNaPUjn7NHkOFgbyBAgV2jaYSRt35BNWuOAA@mail.gmail.com>
 <CAH5HC95rn4gON8zNrFXFY-Cn4z0NZuDU_1TDwKnafX_25_C0UA@mail.gmail.com>
 <20241024025357.a8.nmisch@google.com>
 <CAH5HC97fbvr63wJO23k6N0YmQh8BK3QE8ch3cQC=dWJJBxaCjw@mail.gmail.com>
 <20241031013924.cd.nmisch@google.com>
 <10ec0bc3-5933-1189-6bb8-5dec4114558e@gmail.com>
 <20241031200139.b4@rfd.leadboat.com> <20241101042052.4c.nmisch@google.com>
 <20241101233829.9f.nmisch@google.com> <20241103182925.93.nmisch@google.com>
 <20250510151423.c2.nmisch@google.com>
 <CA+renyU+LGLvCqS0=fHit-N1J-2=2_mPK97AQxvcfKm+F-DxJA@mail.gmail.com>
In-Reply-To: 
 <CA+renyU+LGLvCqS0=fHit-N1J-2=2_mPK97AQxvcfKm+F-DxJA@mail.gmail.com>
From: Paul A Jungwirth <pj@illuminatedcomputing.com>
Date: Sun, 3 Aug 2025 12:32:31 -0700
Message-ID: 
 <CA+renyWSo3WigOO68ydGCQmDdZGLvXFGP3mLGo03XN09AKtDig@mail.gmail.com>
Subject: Re: Inval reliability, especially for inplace updates
To: Noah Misch <noah@leadboat.com>
Cc: Alexander Lakhin <exclusion@gmail.com>,
 Nitin Motiani <nitinmotiani@google.com>,
	Andres Freund <andres@anarazel.de>, Ilyasov Ian <ianilyasov@outlook.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2BrenyWSo3WigOO68ydGCQmDdZGLvXFGP3mLGo03XN09AKtDig%40mail.gmail.com>
Precedence: bulk

On Thu, Jul 31, 2025 at 9:53=E2=80=AFAM Paul A Jungwirth
<pj@illuminatedcomputing.com> wrote:
> On Thu, Oct 31, 2024 at 09:20:52PM -0700, Noah Misch wrote:
> > Here, one of the autovacuum workers had the guilty stack trace, appeari=
ng at
> > the end of this message.  heap_inplace_update_and_unlock() calls
> > CacheInvalidateHeapTupleInplace() while holding BUFFER_LOCK_EXCLUSIVE o=
n a
> > buffer of pg_class.  CacheInvalidateHeapTupleInplace() may call
> > CatalogCacheInitializeCache(), which opens the cache's rel.  If there's=
 not a
> > valid relcache entry for the catcache's rel, we scan pg_class to make a=
 valid
> > relcache entry.  The ensuing hang makes sense.
>
> Personally I never expected that catcache could depend on relcache,
> since it seems lower-level. But it makes sense that you need a
> relcache of pg_class at least, so their relationship is more
> complicated than just layers.
>
> I'm struggling to understand how your explanation incorporates
> *concurrency*, since a self-deadlock only involves locks from one
> backend. But the point is that a concurrent invalidation causes the
> relcache entry to vanish, so that we need to rebuild it. (We can't get
> this far without having built the relcache for pg_class once already.)
>
> Specifically, we drop the table while autovacuum is updating its
> statistics. But how is that possible? Don't both those things
> exclusive-lock the row in pg_class? I must be misunderstanding.

I was curious so I decided to look into this some more. We have a
self-deadlock, but it's rare. Concurrency is necessary to trigger it.
The patch fixes the double-locking, but what was the sequence that
caused the problem?

The only thing reproi.sh does is add & drop tables over and over.

I looked at a few reproductions, and the situation was always the
same: one of the autovacuum workers was stuck analyzing pg_attribute,
with a stack trace like this (no different than Noah's):

#0  __futex_abstimed_wait_common
(futex_word=3Dfutex_word@entry=3D0x7fa76af77238,
expected=3Dexpected@entry=3D0, clockid=3Dclockid@entry=3D0,
    abstime=3Dabstime@entry=3D0x0, private=3D<optimized out>,
cancel=3Dcancel@entry=3Dtrue) at ./nptl/futex-internal.c:103
#1  0x00007fa7742d5f7b in __GI___futex_abstimed_wait_cancelable64
(futex_word=3Dfutex_word@entry=3D0x7fa76af77238,
expected=3Dexpected@entry=3D0,
    clockid=3Dclockid@entry=3D0, abstime=3Dabstime@entry=3D0x0,
private=3D<optimized out>) at ./nptl/futex-internal.c:139
#2  0x00007fa7742e0cff in do_futex_wait (sem=3Dsem@entry=3D0x7fa76af77238,
abstime=3D0x0, clockid=3D0) at ./nptl/sem_waitcommon.c:111
#3  0x00007fa7742e0d90 in __new_sem_wait_slow64
(sem=3Dsem@entry=3D0x7fa76af77238, abstime=3D0x0, clockid=3D0) at
./nptl/sem_waitcommon.c:183
#4  0x00007fa7742e0df9 in __new_sem_wait
(sem=3Dsem@entry=3D0x7fa76af77238) at ./nptl/sem_wait.c:42
#5  0x00005583c641a6a8 in PGSemaphoreLock (sema=3D0x7fa76af77238) at
pg_sema.c:327
#6  0x00005583c652e1da in LWLockAcquire (lock=3D0x7fa76b4fb4e4,
mode=3Dmode@entry=3DLW_SHARED) at lwlock.c:1318
#7  0x00005583c64df801 in LockBuffer (buffer=3D36, mode=3Dmode@entry=3D1) a=
t
bufmgr.c:4182
#8  0x00005583c5eda4b3 in heapam_index_fetch_tuple
(scan=3D0x7fa76adc1b30, tid=3D0x7fa76adc3fb8, snapshot=3D0x5583e8ede3d8,
slot=3D0x7fa76adc1030,
    call_again=3D0x7fa76adc3fbe, all_dead=3D0x7ffebf643e6f) at
heapam_handler.c:146
#9  0x00005583c5efb548 in table_index_fetch_tuple (scan=3D<optimized
out>, tid=3Dtid@entry=3D0x7fa76adc3fb8, snapshot=3D<optimized out>,
    slot=3Dslot@entry=3D0x7fa76adc1030,
call_again=3Dcall_again@entry=3D0x7fa76adc3fbe,
all_dead=3Dall_dead@entry=3D0x7ffebf643e6f)
    at ../../../../src/include/access/tableam.h:1226
#10 0x00005583c5efd1a7 in index_fetch_heap
(scan=3Dscan@entry=3D0x7fa76adc3f58, slot=3Dslot@entry=3D0x7fa76adc1030) at
indexam.c:622
#11 0x00005583c5efd362 in index_getnext_slot (scan=3D0x7fa76adc3f58,
direction=3Ddirection@entry=3DForwardScanDirection, slot=3D0x7fa76adc1030)
    at indexam.c:682
#12 0x00005583c5efa32a in systable_getnext
(sysscan=3Dsysscan@entry=3D0x7fa76adc3dd0) at genam.c:512
#13 0x00005583c677fd20 in ScanPgRelation (targetRelId=3D<optimized out>,
indexOK=3DindexOK@entry=3Dtrue,
    force_non_historic=3Dforce_non_historic@entry=3Dfalse) at
relcache.c:385
#14 0x00005583c6780cf8 in RelationReloadNailed
(relation=3Drelation@entry=3D0x7fa7740fd698) at relcache.c:2381
#15 0x00005583c678b67e in RelationClearRelation
(relation=3Drelation@entry=3D0x7fa7740fd698, rebuild=3D<optimized out>) at
relcache.c:2527
#16 0x00005583c678ce85 in RelationFlushRelation
(relation=3D0x7fa7740fd698) at relcache.c:2839
#17 0x00005583c678cf00 in RelationCacheInvalidateEntry
(relationId=3D<optimized out>) at relcache.c:2900
#18 0x00005583c67714fe in LocalExecuteInvalidationMessage
(msg=3D0x7ffebf644220) at inval.c:666
#19 0x00005583c651585b in ReceiveSharedInvalidMessages (
    invalFunction=3DinvalFunction@entry=3D0x5583c677130d
<LocalExecuteInvalidationMessage>,
    resetFunction=3DresetFunction@entry=3D0x5583c67700cf
<InvalidateSystemCaches>) at sinval.c:121
#20 0x00005583c677012c in AcceptInvalidationMessages () at inval.c:766
#21 0x00005583c651f8d2 in LockRelationOid (relid=3D<optimized out>,
lockmode=3D1) at lmgr.c:137
#22 0x00005583c5e503db in relation_open
(relationId=3DrelationId@entry=3D1259, lockmode=3Dlockmode@entry=3D1) at
relation.c:56
#23 0x00005583c5f595e5 in table_open
(relationId=3DrelationId@entry=3D1259, lockmode=3Dlockmode@entry=3D1) at
table.c:43
#24 0x00005583c677fc8c in ScanPgRelation (targetRelId=3D<optimized out>,
indexOK=3D<optimized out>,
    force_non_historic=3Dforce_non_historic@entry=3Dfalse) at
relcache.c:368
#25 0x00005583c67807c1 in RelationReloadIndexInfo
(relation=3Drelation@entry=3D0x7fa774141028) at relcache.c:2257
#26 0x00005583c6780dff in RelationReloadNailed
(relation=3Drelation@entry=3D0x7fa774141028) at relcache.c:2359
#27 0x00005583c678b67e in RelationClearRelation
(relation=3Drelation@entry=3D0x7fa774141028, rebuild=3D<optimized out>) at
relcache.c:2527
#28 0x00005583c678ce85 in RelationFlushRelation
(relation=3D0x7fa774141028) at relcache.c:2839
#29 0x00005583c678cf00 in RelationCacheInvalidateEntry
(relationId=3D<optimized out>) at relcache.c:2900
#30 0x00005583c67714fe in LocalExecuteInvalidationMessage
(msg=3D0x7ffebf644640) at inval.c:666
#31 0x00005583c651585b in ReceiveSharedInvalidMessages (
    invalFunction=3DinvalFunction@entry=3D0x5583c677130d
<LocalExecuteInvalidationMessage>,
    resetFunction=3DresetFunction@entry=3D0x5583c67700cf
<InvalidateSystemCaches>) at sinval.c:121
#32 0x00005583c677012c in AcceptInvalidationMessages () at inval.c:766
#33 0x00005583c651f8d2 in LockRelationOid (relid=3D<optimized out>,
lockmode=3D1) at lmgr.c:137
#34 0x00005583c5e503db in relation_open
(relationId=3DrelationId@entry=3D1259, lockmode=3Dlockmode@entry=3D1) at
relation.c:56
#35 0x00005583c5f595e5 in table_open
(relationId=3DrelationId@entry=3D1259, lockmode=3Dlockmode@entry=3D1) at
table.c:43
#36 0x00005583c677fc8c in ScanPgRelation (targetRelId=3D<optimized out>,
indexOK=3DindexOK@entry=3Dtrue,
    force_non_historic=3Dforce_non_historic@entry=3Dfalse) at relcache.c:36=
8
#37 0x00005583c6780cf8 in RelationReloadNailed
(relation=3Drelation@entry=3D0x7fa7740fd698) at relcache.c:2381
#38 0x00005583c678b67e in RelationClearRelation
(relation=3Drelation@entry=3D0x7fa7740fd698, rebuild=3D<optimized out>) at
relcache.c:2527
#39 0x00005583c678ce85 in RelationFlushRelation
(relation=3D0x7fa7740fd698) at relcache.c:2839
#40 0x00005583c678cf00 in RelationCacheInvalidateEntry
(relationId=3D<optimized out>) at relcache.c:2900
#41 0x00005583c67714fe in LocalExecuteInvalidationMessage
(msg=3D0x7ffebf644a40) at inval.c:666
#42 0x00005583c651585b in ReceiveSharedInvalidMessages (
    invalFunction=3DinvalFunction@entry=3D0x5583c677130d
<LocalExecuteInvalidationMessage>,
    resetFunction=3DresetFunction@entry=3D0x5583c67700cf
<InvalidateSystemCaches>) at sinval.c:121
#43 0x00005583c677012c in AcceptInvalidationMessages () at inval.c:766
#44 0x00005583c651f8d2 in LockRelationOid (relid=3D<optimized out>,
lockmode=3D1) at lmgr.c:137
#45 0x00005583c5e503db in relation_open (relationId=3D1259,
lockmode=3Dlockmode@entry=3D1) at relation.c:56
#46 0x00005583c5f595e5 in table_open (relationId=3D<optimized out>,
lockmode=3Dlockmode@entry=3D1) at table.c:43
#47 0x00005583c676b11a in CatalogCacheInitializeCache
(cache=3Dcache@entry=3D0x5583e8e79f80) at catcache.c:929
#48 0x00005583c676e3c2 in PrepareToInvalidateCacheTuple
(relation=3Drelation@entry=3D0x7fa7740fd698,
tuple=3Dtuple@entry=3D0x7fa76adc3a88,
    newtuple=3Dnewtuple@entry=3D0x0,
function=3Dfunction@entry=3D0x5583c676fac8 <RegisterCatcacheInvalidation>,
    context=3Dcontext@entry=3D0x7fa76adc38b8) at catcache.c:2164
#49 0x00005583c676fdc3 in CacheInvalidateHeapTupleCommon
(relation=3Drelation@entry=3D0x7fa7740fd698,
tuple=3Dtuple@entry=3D0x7fa76adc3a88,
    newtuple=3Dnewtuple@entry=3D0x0,
prepare_callback=3Dprepare_callback@entry=3D0x5583c676fe93
<PrepareInplaceInvalidationState>) at inval.c:1350
#50 0x00005583c6770ab9 in CacheInvalidateHeapTupleInplace
(relation=3Drelation@entry=3D0x7fa7740fd698,
tuple=3Dtuple@entry=3D0x7fa76adc3a88,
    newtuple=3Dnewtuple@entry=3D0x0) at inval.c:1461
#51 0x00005583c5ec8dc9 in heap_inplace_update_and_unlock
(relation=3D0x7fa7740fd698, oldtup=3D<optimized out>,
tuple=3D0x7fa76adc3a88, buffer=3D36)
    at heapam.c:6311
#52 0x00005583c5efb33a in systable_inplace_update_finish
(state=3D0x7fa76adc3748, tuple=3D<optimized out>) at genam.c:884
#53 0x00005583c61fbc92 in vac_update_relstats
(relation=3Drelation@entry=3D0x7fa774102b20, num_pages=3Dnum_pages@entry=3D=
89,
num_tuples=3D4460,
    num_all_visible_pages=3D<optimized out>,
hasindex=3Dhasindex@entry=3Dtrue, frozenxid=3Dfrozenxid@entry=3D0,
minmulti=3Dminmulti@entry=3D0,
    frozenxid_updated=3D0x0, minmulti_updated=3D0x0, in_outer_xact=3Dfalse)
at vacuum.c:1455
#54 0x00005583c60ec3f2 in do_analyze_rel
(onerel=3Donerel@entry=3D0x7fa774102b20,
params=3Dparams@entry=3D0x5583e8e9bda4,
    va_cols=3Dva_cols@entry=3D0x0, acquirefunc=3D<optimized out>,
relpages=3D89, inh=3Dinh@entry=3Dfalse, in_outer_xact=3Dfalse, elevel=3D13)
    at analyze.c:645
#55 0x00005583c60ed155 in analyze_rel (relid=3D<optimized out>,
relation=3D0x5583e8ef9fa0, params=3Dparams@entry=3D0x5583e8e9bda4,
va_cols=3D0x0,
    in_outer_xact=3D<optimized out>, bstrategy=3D<optimized out>) at analyz=
e.c:262
#56 0x00005583c61fd2d4 in vacuum (relations=3D0x5583e8efc008,
params=3Dparams@entry=3D0x5583e8e9bda4, bstrategy=3D<optimized out>,
    bstrategy@entry=3D0x5583e8ef2e00, isTopLevel=3DisTopLevel@entry=3Dtrue)
at vacuum.c:493
#57 0x00005583c641e58b in autovacuum_do_vac_analyze
(tab=3Dtab@entry=3D0x5583e8e9bda0,
bstrategy=3Dbstrategy@entry=3D0x5583e8ef2e00)
    at autovacuum.c:3180
#58 0x00005583c6420f84 in do_autovacuum () at autovacuum.c:2503
#59 0x00005583c64219ae in AutoVacWorkerMain (argc=3Dargc@entry=3D0,
argv=3Dargv@entry=3D0x0) at autovacuum.c:1716
#60 0x00005583c6421bb3 in StartAutoVacWorker () at autovacuum.c:1494
#61 0x00005583c6433c22 in StartAutovacuumWorker () at postmaster.c:5536
#62 0x00005583c64344ca in sigusr1_handler
(postgres_signal_arg=3D<optimized out>) at postmaster.c:5241
#63 <signal handler called>
#64 0x00007fa77434e904 in __GI___select (nfds=3Dnfds@entry=3D8,
readfds=3Dreadfds@entry=3D0x7ffebf6461c0, writefds=3Dwritefds@entry=3D0x0,
    exceptfds=3Dexceptfds@entry=3D0x0,
timeout=3Dtimeout@entry=3D0x7ffebf6461b0) at
../sysdeps/unix/sysv/linux/select.c:69
#65 0x00005583c6435442 in ServerLoop () at postmaster.c:1773
#66 0x00005583c6437c5e in PostmasterMain (argc=3Dargc@entry=3D1,
argv=3Dargv@entry=3D0x5583e8e0fe00) at postmaster.c:1481
#67 0x00005583c62d57f5 in main (argc=3D1, argv=3D0x5583e8e0fe00) at main.c:=
202

In frame 55 we are analyzing relation 1249 (pg_attribute):

(gdb) f 55
#55 0x00005583c60ed155 in analyze_rel (relid=3D<optimized out>,
relation=3D0x5583e8ef9fa0, params=3Dparams@entry=3D0x5583e8e9bda4,
va_cols=3D0x0,
    in_outer_xact=3D<optimized out>, bstrategy=3D<optimized out>) at analyz=
e.c:262
262                     do_analyze_rel(onerel, params, va_cols, acquirefunc=
,
(gdb) p *relation
$1 =3D {type =3D T_RangeVar, catalogname =3D 0x0, schemaname =3D
0x5583e8ef3308 "pg_catalog", relname =3D 0x5583e8ef32e0 "pg_attribute",
inh =3D true,
  relpersistence =3D 112 'p', alias =3D 0x0, location =3D -1}
(gdb) p *onerel
$2 =3D {rd_node =3D {spcNode =3D 1663, dbNode =3D 5, relNode =3D 1249}, rd_=
smgr
=3D 0x5583e8eb98c0, rd_refcnt =3D 2, rd_backend =3D -1,
  rd_islocaltemp =3D false, rd_isnailed =3D true, rd_isvalid =3D true,
rd_indexvalid =3D true, rd_statvalid =3D false, rd_createSubid =3D 0,
  rd_newRelfilenodeSubid =3D 0, rd_firstRelfilenodeSubid =3D 0,
rd_droppedSubid =3D 0, rd_rel =3D 0x7fa774102d38, rd_att =3D 0x7fa774102e50=
,
  rd_id =3D 1249, rd_lockInfo =3D {lockRelId =3D {relId =3D 1249, dbId =3D =
5}},
rd_rules =3D 0x0, rd_rulescxt =3D 0x0, trigdesc =3D 0x0, rd_rsdesc =3D 0x0,
  rd_fkeylist =3D 0x0, rd_fkeyvalid =3D false, rd_partkey =3D 0x0,
rd_partkeycxt =3D 0x0, rd_partdesc =3D 0x0, rd_pdcxt =3D 0x0,
  rd_partdesc_nodetached =3D 0x0, rd_pddcxt =3D 0x0,
rd_partdesc_nodetached_xmin =3D 0, rd_partcheck =3D 0x0, rd_partcheckvalid
=3D false,
  rd_partcheckcxt =3D 0x0, rd_indexlist =3D 0x7fa774120298, rd_pkindex =3D
2659, rd_replidindex =3D 0, rd_statlist =3D 0x0, rd_indexattr =3D 0x0,
  rd_keyattr =3D 0x0, rd_pkattr =3D 0x0, rd_idattr =3D 0x0, rd_pubdesc =3D
0x0, rd_options =3D 0x0, rd_amhandler =3D 3,
  rd_tableam =3D 0x5583c6b0bf00 <heapam_methods>, rd_index =3D 0x0,
rd_indextuple =3D 0x0, rd_indexcxt =3D 0x0, rd_indam =3D 0x0, rd_opfamily =
=3D
0x0,
  rd_opcintype =3D 0x0, rd_support =3D 0x0, rd_supportinfo =3D 0x0,
rd_indoption =3D 0x0, rd_indexprs =3D 0x0, rd_indpred =3D 0x0, rd_exclops =
=3D
0x0,
  rd_exclprocs =3D 0x0, rd_exclstrats =3D 0x0, rd_indcollation =3D 0x0,
rd_opcoptions =3D 0x0, rd_amcache =3D 0x0, rd_fdwroutine =3D 0x0,
  rd_toastoid =3D 0, pgstat_enabled =3D true, pgstat_info =3D 0x5583e8eb009=
0}

It makes sense that pg_attribute would experience a lot of churn,
since we just add & drop tables.

We call vac_update_relstats, which does an in-place update to pg_class
(frame 53).

Within vac_update_relstats, we call system_inplace_update_begin, which
LWLocks the buffer we want to update. That's the first part of the
self-deadlock. But it's no problem yet. We proceed in
vac_update_relstats to sys_inplace_update_finish. . . .

That sends an inplace invalidation message for the pg_class tuple
(frame 50). So we need to replace the tuple in the pg_class catcache.
But before we do that, we make sure the catcache is initialized (frame
47):

    /* Just in case cache hasn't finished initialization yet... */
    if (ccp->cc_tupdesc =3D=3D NULL)
        CatalogCacheInitializeCache(ccp);

That locks pg_class with AccessShareLock, which looks for pending
invalidation messages (frame 43).

It finds one invalidating the relcache for pg_class:

(gdb) f 41
#41 0x00005583c67714fe in LocalExecuteInvalidationMessage
(msg=3D0x7ffebf644a40) at inval.c:666
666
RelationCacheInvalidateEntry(msg->rc.relId);
(gdb) p *msg
$9 =3D {id =3D -2 '\376', cc =3D {id =3D -2 '\376', dbId =3D 5, hashValue =
=3D
1259}, cat =3D {id =3D -2 '\376', dbId =3D 5, catId =3D 1259}, rc =3D {
    id =3D -2 '\376', dbId =3D 5, relId =3D 1259}, sm =3D {id =3D -2 '\376'=
,
backend_hi =3D -47 '\321', backend_lo =3D 59642, rnode =3D {spcNode =3D 5,
      dbNode =3D 1259, relNode =3D 32679}}, rm =3D {id =3D -2 '\376', dbId =
=3D
5}, sn =3D {id =3D -2 '\376', dbId =3D 5, relId =3D 1259}}

(Btw where did this message come from?)

So we have to rebuild the relcache for pg_class. Obviously we need to
look up its entry in (heh) pg_class (frame 37). So we open that table
with an AccessShareLock, which looks for pending invalidation messages
(frame 32). We find one telling us to invalidate 2662
(pg_class_oid_index):

(gdb) f 30
#30 0x00005583c67714fe in LocalExecuteInvalidationMessage
(msg=3D0x7ffebf644640) at inval.c:666
666
RelationCacheInvalidateEntry(msg->rc.relId);
(gdb) p *msg
$12 =3D {id =3D -2 '\376', cc =3D {id =3D -2 '\376', dbId =3D 5, hashValue =
=3D
2662}, cat =3D {id =3D -2 '\376', dbId =3D 5, catId =3D 2662}, rc =3D {
    id =3D -2 '\376', dbId =3D 5, relId =3D 2662}, sm =3D {id =3D -2 '\376'=
,
backend_hi =3D -47 '\321', backend_lo =3D 59642, rnode =3D {spcNode =3D 5,
      dbNode =3D 2662, relNode =3D 32679}}, rm =3D {id =3D -2 '\376', dbId =
=3D
5}, sn =3D {id =3D -2 '\376', dbId =3D 5, relId =3D 2662}}

I guess that is coming from the add & drop table commands.

So we reload the relcache entry for 2662 (frame 26). That opens
pg_class with AccessShareLock, which looks for pending invalidation
messages (frame 20). We find one telling us to invalidate pg_class:

(gdb) f 18
#18 0x00005583c67714fe in LocalExecuteInvalidationMessage
(msg=3D0x7ffebf644220) at inval.c:666
666
RelationCacheInvalidateEntry(msg->rc.relId);
(gdb) p *msg
$14 =3D {id =3D -2 '\376', cc =3D {id =3D -2 '\376', dbId =3D 5, hashValue =
=3D
1259}, cat =3D {id =3D -2 '\376', dbId =3D 5, catId =3D 1259}, rc =3D {
    id =3D -2 '\376', dbId =3D 5, relId =3D 1259}, sm =3D {id =3D -2 '\376'=
,
backend_hi =3D 75 'K', backend_lo =3D 48996, rnode =3D {spcNode =3D 5,
      dbNode =3D 1259, relNode =3D 32679}}, rm =3D {id =3D -2 '\376', dbId =
=3D
5}, sn =3D {id =3D -2 '\376', dbId =3D 5, relId =3D 1259}}

So we rebuild the relcache for pg_class, which means scanning pg_class
(frame 14). We try to lock one of its buffers (frame 7), and we
self-deadlock.

So the problem is triggered by an invalidation message for the
pg_class relcache entry. But what sends that message? (Actually it
seems we got two of them.) Why does anything we're doing invalidate
pg_class? Normally an inplace update to pg_class sends an invalidation
message for the relation described by that tuple, not all of pg_class
itself (see CacheInvalidateHeapTupleCommon).

I couldn't figure that out, until I put a breakpoint at the top of
RegisterRelcacheInvalidation and ran `vacuum analyze`. When we analyze
pg_class (and btw the repro script gives it a lot of churn), we call
vac_update_relstats on it, so we are doing an inplace update of
pg_class on the tuple *about* pg_class. So that's why we send the
invalidation about the whole table.

The concurrency, then, is analyzing pg_attribute in one worker and
pg_class in another. No locks would prevent that. It doesn't have to
be pg_attribute; it could be some user table. But since the repro
script adds & drops tables, pg_attribute is getting churned.

Yours,

--=20
Paul              ~{:-)
pj@illuminatedcomputing.com