From: Michael Paquier <michael@paquier.xyz>
To: Floris Van Nee <florisvannee@optiver.com>
Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: tgl@sss.pgh.pa.us <tgl@sss.pgh.pa.us>
Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: error "can only drop stats once" brings down database
Date: Mon, 10 Jun 2024 15:37:24 +0900
Message-ID: <ZmafJO3lpMjbyHXv@paquier.xyz> (raw)
In-Reply-To: <2cb577e176774caca28c724898768a38@Optiver.com>
References: <68430b1b76d44558b7b07ae8c69151f6@Optiver.com>
<ZmE-qm8d9kfDfnMy@paquier.xyz>
<ZmFRlwr/53/tXqxn@ip-10-97-1-34.eu-west-3.compute.internal>
<20240607.101540.1972690976596948719.horikyota.ntt@gmail.com>
<ZmJ0bBODm6AitiC3@paquier.xyz>
<20240607031258.l4fcpvulo3uee5g6@awork3.anarazel.de>
<ZmLTneAb+EaZMS9A@ip-10-97-1-34.eu-west-3.compute.internal>
<ZmLXnAtRSUudsFqg@paquier.xyz>
<2cb577e176774caca28c724898768a38@Optiver.com>
On Sat, Jun 08, 2024 at 11:52:43AM +0000, Floris Van Nee wrote:
> I've got an update about the bug. I managed to reproduce it locally
> after a lot of digging.
>
> How to repro:
> - Setup primary + replica
> - Open a psql session on both
> - On primary session: create table t (a int); select 't'::regclass::oid;
> - On replica session: select * from t;
> - On primary session: drop table t; vacuum pg_class; checkpoint;
> - Gdb attach to the backend for your primary, set a breakpoint for
> catalog.c:GetNewOidWithIndex, just before it calls GetNewObjectId()
> - On primary session: create table t (a int);
> - When it hits breakpoint, simulate oid wraparound by setting:
> ShmemVariableCache->nextOid = <the output value of the select earlier>
> This will make pg create the new table with the same oid as the previous one.
> - On primary session: drop table t; -- this triggers the replica to go down
Okay, this stuff makes the beginning of a week fun.
> The reason it crashes on replica is that the recovery process is responsible for dropping
> stats on commit, but it's not creating them on table creation. Thus, on the second create
> table call, the old shared stats entry still exists (due to a backend still have a ref to it),
> but it is never reinitialized by the logic in pgstat_reinit_entry(). On primary it's not possible
> to reach this state, because heap_create() creates the stats entry immediately when the
> table is created.
>
> I wonder what's the best way to fix this though.
> Should redo process call pgstat_create_relation somewhere, just like heap_create does?
> Should we just ignore this 'drop stats twice' error on standby?
Nah, ignoring the double-drop error does not seem right to me.
Wouldn't it make the most sense to ensure that the stats are dropped
on the standby instead on the first DROP replayed even if there are
still references to it hold, making sure that the stats entry with
this OID is gone before reusing it after wraparound?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: michael@paquier.xyz, florisvannee@optiver.com, bertranddrouvot.pg@gmail.com, andres@anarazel.de, horikyota.ntt@gmail.com, tgl@sss.pgh.pa.us, pgsql-bugs@lists.postgresql.org
Subject: Re: error "can only drop stats once" brings down database
In-Reply-To: <ZmafJO3lpMjbyHXv@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox