Subject: BUG #19520: PANIC when concurrently manipulating stored procedures with pg_stat_statements and track_functions =

From: PG Bug reporting form <[email-masked]>
Date: Sun, 14 Jun 2026 16:05:27 +0000

The following bug has been logged on the website:

Bug reference:      19520
Logged by:          zhanglihui
Email address:      [email-masked]
PostgreSQL version: 19beta1
Operating system:   Ubuntu 25.04
Description:        

=== Configuration ===
Enabled extensions: pg_stat_statements
postgresql.conf:
shared_preload_libraries = 'pg_stat_statements'
track_functions = 'all'

=== Problem Description ===
PostgreSQL server throws PANIC under high concurrent create, CALL and DROP
of stored procedures.
This issue **only reproduces when pg_stat_statements is enabled and
track_functions = all**.
It cannot be triggered if pg_stat_statements is disabled or track_functions
is set to none/pl.

=== Run Steps ===
javac -cp postgresql-42.7.5.jar -d out     src/ConcurrentSqlTest.java
src/ProcCrashReprodure.java
# Terminal 1 — Mixed DDL/DML load test (40 threads, 10000 iterations each)
java -cp out:postgresql-42.7.5.jar ConcurrentSqlTest

# Terminal 2 — Pure CALL load test (20 threads, infinite loop)
java -cp out:postgresql-42.7.5.jar ProcCrashReprodure
# Note: If no PANIC log or core dump is generated after the execution of
Terminal 1, please re-run the command repeatedly until the issue occurs.

PANIC log：
postgresql-2026-06-14_235441.log:2026-06-14 23:54:41.949 CST [691761] PANIC:
XX000: cannot abort transaction 4166281, it was already committed
postgresql-2026-06-14_235931.log:2026-06-14 23:59:31.556 CST [696980] PANIC:
XX000: cannot abort transaction 4641943, it was already committed

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=6,
no_tid=0) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (threadid=<optimized out>, signo=6) at
./nptl/pthread_kill.c:89
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at
./nptl/pthread_kill.c:100
#3  0x000073e62ec4579e in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
#4  0x000073e62ec288cd in __GI_abort () at ./stdlib/abort.c:73
#5  0x000056aeb574138c in errfinish (filename=0x56aeb57ff545 "xact.c",
lineno=1835, funcname=0x56aeb5800d00 <__func__.26> "RecordTransactionAbort")
at elog.c:621
#6  0x000056aeb4faae67 in RecordTransactionAbort (isSubXact=false) at
xact.c:1835
#7  0x000056aeb4fac19f in AbortTransaction () at xact.c:2982
#8  0x000056aeb4facc22 in AbortCurrentTransactionInternal () at xact.c:3553
#9  0x000056aeb4facb93 in AbortCurrentTransaction () at xact.c:3507
#10 0x000056aeb5525552 in PostgresMain (dbname=0x56aedecf30d0 "postgres",
username=0x56aedecf30b0 "zlh_user") at postgres.c:4539
#11 0x000056aeb551b59c in BackendMain (startup_data=0x7fffb9501000,
startup_data_len=24) at backend_startup.c:124
#12 0x000056aeb5405686 in postmaster_child_launch (child_type=B_BACKEND,
child_slot=24, startup_data=0x7fffb9501000, startup_data_len=24,
client_sock=0x7fffb9501060) at launch_backend.c:268
#13 0x000056aeb540c11b in BackendStartup (client_sock=0x7fffb9501060) at
postmaster.c:3627
#14 0x000056aeb540969f in ServerLoop () at postmaster.c:1728
#15 0x000056aeb5408f7c in PostmasterMain (argc=1, argv=0x56aedeca1430) at
postmaster.c:1415
#16 0x000056aeb528ce51 in main (argc=1, argv=0x56aedeca1430) at main.c:231

---- File 1: ConcurrentSqlTest.java (Mixed DDL/DML test, 40 threads × 10000
iterations) ----
import java.sql.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * PostgreSQL concurrent DDL/DML stress test — reproduces backend crash
 * (signal 5 / SIGTRAP) when DROP/CREATE PROCEDURE and CALL PROCEDURE
 * execute concurrently at high concurrency.
 *
 * Compile:  javac -cp postgresql-42.7.5.jar ConcurrentSqlTest.java
 * Run:      java -cp .:postgresql-42.7.5.jar ConcurrentSqlTest
 *
 * Optional env vars: PG_HOST, PG_PORT, PG_DATABASE, PG_USER, PG_PASSWORD
 */
public final class ConcurrentSqlTest {

    // ── connection defaults ──
    private static final String HOST     = envOr("PG_HOST",
"192.168.239.128");
    private static final String PORT     = envOr("PG_PORT",     "5432");
    private static final String DATABASE = envOr("PG_DATABASE", "postgres");
    private static final String USER     = envOr("PG_USER",     "zlh_user");
    private static final String PASSWORD = envOr("PG_PASSWORD",
"Gauss@123");
    private static final String JDBC_URL =
            "jdbc:postgresql://" + HOST + ":" + PORT + "/" + DATABASE;

    // ── test parameters ──
    private static final int THREADS    = 40;
    private static final int ITERATIONS = 10_000;

    // ── SQL batches ──
    private static final String[] SQL = {
        "DROP PROCEDURE IF EXISTS proc_test",
        "CREATE OR REPLACE PROCEDURE proc_test()\n"
                + "LANGUAGE plpgsql\n"
                + "AS $$\n"
                + "BEGIN\n"
                + "END;\n"
                + "$$",
        "CALL proc_test()",
    };

    // ── counters ──
    private static final AtomicInteger totalOk   = new AtomicInteger(0);
    private static final AtomicInteger totalFail = new AtomicInteger(0);

    static {
        try { Class.forName("org.postgresql.Driver"); }
        catch (ClassNotFoundException e) {
            System.err.println("PostgreSQL JDBC driver not found on
classpath");
            System.exit(1);
        }
    }

    public static void main(String[] args) throws InterruptedException {
        System.out.println("=== PostgreSQL concurrent DROP/CREATE/CALL
PROCEDURE ===");
        System.out.println("URL: " + JDBC_URL);
        System.out.println("Threads: " + THREADS + " | Iterations/thread: "
+ ITERATIONS);
        System.out.println("Watch for: backend terminated by signal 5
(SIGTRAP)");
System.out.println("========================================================");

        long t0 = System.currentTimeMillis();
        ExecutorService pool = Executors.newFixedThreadPool(THREADS);
        List<Future<?>> futures = new ArrayList<>();

        for (int i = 0; i < THREADS; i++) {
            final int tid = i;
            futures.add(pool.submit(() -> runWorker(tid)));
        }

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            System.out.println("\nShutting down...");
            pool.shutdownNow();
        }));

        // drain
        for (Future<?> f : futures) {
            try { f.get(); } catch (ExecutionException e) {
                System.err.println("Thread crashed: " +
e.getCause().getMessage());
            } catch (CancellationException ignored) { }
        }
        pool.shutdown();

        long elapsed = System.currentTimeMillis() - t0;
System.out.println("========================================================");
        System.out.printf("Done.  OK: %d  FAIL: %d  Time: %.1fs%n",
                totalOk.get(), totalFail.get(), elapsed / 1000.0);
        System.exit(totalFail.get() > 0 ? 1 : 0);
    }

    // ── single worker (reuses one connection) ──
    private static void runWorker(int tid) {
        int ok = 0, fail = 0;
        Connection c = openConnection();
        for (int i = 0; i < ITERATIONS; i++) {
            if (Thread.currentThread().isInterrupted()) break;
            try {
                for (String sql : SQL) {
                    try (Statement s = c.createStatement()) {
                        s.execute(sql);
                    }
                }
                ok++;
            } catch (SQLException e) {
                fail++;
                if (totalFail.incrementAndGet() <= 10) {
                    System.err.printf("Thread %d loop %d: [%s] %s%n",
                            tid, i, e.getSQLState(),
e.getMessage().replace('\n', ' '));
                }
                // reconnect if server closed the connection (e.g. after
crash)
                try {
                    if (c.isClosed()) c = openConnection();
                } catch (SQLException ignored) { }
            }
        }
        totalOk.addAndGet(ok);
        close(c);
        System.out.printf("Thread %2d: %d ok / %d fail%n", tid, ok, fail);
    }

    private static Connection openConnection() {
        try {
            Connection c = DriverManager.getConnection(JDBC_URL, USER,
PASSWORD);
            c.setAutoCommit(true);
            return c;
        } catch (SQLException e) {
            throw new RuntimeException("Failed to connect: " +
e.getMessage(), e);
        }
    }

    private static void close(Connection c) {
        try { if (c != null) c.close(); } catch (SQLException ignored) { }
    }

    private static String envOr(String key, String def) {
        String v = System.getenv(key);
        return (v != null && !v.trim().isEmpty()) ? v : def;
    }
}

---- File 2: ProcCrashReprodure.java (Pure CALL test, 20 threads, infinite
loop) ----
import java.sql.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicBoolean;

/**
 * PostgreSQL concurrent CALL stress test — run this together with
 * ConcurrentSqlTest to reproduce backend crash (signal 5 / SIGTRAP).
 *
 * This program does ONLY repeated CALL proc_test() across 20 threads
 * (infinite loop).  ConcurrentSqlTest does DROP → CREATE → CALL in
 * a loop.  Run both simultaneously.
 *
 * Compile:  javac -cp postgresql-42.7.5.jar ProcCrashReprodure.java
 * Run:      java -cp .:postgresql-42.7.5.jar ProcCrashReprodure
 *
 * Optional env vars: PG_HOST, PG_PORT, PG_DATABASE, PG_USER, PG_PASSWORD
 */
public final class ProcCrashReprodure {

    private static final String HOST     = envOr("PG_HOST",
"192.168.239.128");
    private static final String PORT     = envOr("PG_PORT",     "5432");
    private static final String DATABASE = envOr("PG_DATABASE", "postgres");
    private static final String USER     = envOr("PG_USER",     "zlh_user");
    private static final String PASSWORD = envOr("PG_PASSWORD",
"Gauss@123");
    private static final String JDBC_URL =
            "jdbc:postgresql://" + HOST + ":" + PORT + "/" + DATABASE;

    private static final int THREADS = 20;

    private static final AtomicBoolean stop = new AtomicBoolean(false);

    static {
        try { Class.forName("org.postgresql.Driver"); }
        catch (ClassNotFoundException e) {
            System.err.println("PostgreSQL JDBC driver not found on
classpath");
            System.exit(1);
        }
    }

    public static void main(String[] args) {
        System.out.println("=== PostgreSQL CALL stress test (run
ConcurrentSqlTest too) ===");
        System.out.println("URL: " + JDBC_URL);
        System.out.println("Threads: " + THREADS + "  |  Loop: infinite
(Ctrl+C to stop)");
        System.out.println("Watch for: backend terminated by signal 5
(SIGTRAP)");
System.out.println("================================================================");

        ExecutorService pool = Executors.newFixedThreadPool(THREADS);
        CountDownLatch latch = new CountDownLatch(THREADS);

        for (int i = 0; i < THREADS; i++) {
            final int tid = i;
            pool.submit(() -> {
                try {
                    runWorker(tid);
                } finally {
                    latch.countDown();
                }
            });
        }

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            System.out.println("\nShutting down...");
            stop.set(true);
            pool.shutdownNow();
        }));

        try { latch.await(); }
        catch (InterruptedException e) { Thread.currentThread().interrupt();
}

        pool.shutdownNow();
        System.out.println("Done.");
    }

    private static void runWorker(int tid) {
        while (!stop.get()) {
            try (Connection c = DriverManager.getConnection(JDBC_URL, USER,
PASSWORD)) {
                c.setAutoCommit(true);
                while (!stop.get()) {
                    try (CallableStatement cs = c.prepareCall("{ CALL
proc_test() }")) {
                        cs.execute();
                    } catch (SQLException e) {
                        // expected when proc is being dropped/recreated
                        if (c.isClosed()) break;  // reconnect
                    }
                }
            } catch (SQLException e) {
                // connection failure — pause then reconnect
                try { Thread.sleep(100); }
                catch (InterruptedException ie) { break; }
            }
        }
    }

    private static String envOr(String key, String def) {
        String v = System.getenv(key);
        return (v != null && !v.trim().isEmpty()) ? v : def;
    }
}

---

From: Ayush Tiwari <[email-masked]>
Date: Mon, 15 Jun 2026 14:44:06 +0530

Hi,

> #16 0x000056aeb528ce51 in main (argc=1, argv=0x56aedeca1430) at main.c:231

Thanks for the report!  I'm unsure if this has already been reported or
not.

I looked into this the last day, I could reproduce it locally.  Rather
than the Java harness I used ~60 concurrent psql clients looping DROP /
CREATE OR REPLACE / CALL of the same empty plpgsql procedure
(track_functions=all, pg_stat_statements loaded); here it PANICs within
a few seconds.

Just before the PANIC the failing backend logs:

  ERROR:  trying to drop stats entry already dropped: kind=function ...
  WARNING:  AbortTransaction while in COMMIT state
  PANIC:  cannot abort transaction xxx, it was already committed

So it looks like a function's shared stats entry gets dropped twice:
once out-of-band from pgstat_init_function_usage() when a concurrent
CALL notices the function is gone, and once from the transactional drop
at DROP time.  When the latter loses the race it runs from
AtEOXact_PgStat(), past the commit record, so the "already dropped"
elog() in pgstat_drop_entry_internal() becomes the PANIC.

The two droppers and the guard all seem to date back to PG 15
(5891c7a8ed8f).  I guess the "drop exactly once" assumption behind that
guard doesn't really hold for function stats, where two independent
droppers are legitimate.

I've added Andres and Michael on the thread, since they have worked on
this in the past, for their input.

Regards,
Ayush

---

From: Michael Paquier <[email-masked]>
Date: Tue, 16 Jun 2026 08:24:51 +0900

> this in the past, for their input.

Thanks for the poke.  I have marked this thread as something to look
at, but was not able to get back to it.  Will investigate..
--
Michael

---

From: Michael Paquier <[email-masked]>
Date: Wed, 17 Jun 2026 13:15:13 +0900

> at, but was not able to get back to it.  Will investigate..

As far as I can see, pgss is not really a requirement.  Your case is
taking advantage of the module introducing more slowness to enlarge
the reproduction window.  Now saying that pgss being slow is a good
thing, it's bad, but it helps here.  I've tried to reproduce in three
environments, only my mac is able to get something, because it's
slower I guess..

Attached is a script able to reproduce the issue in bash, courtesy of
Claude because java and I sum up to a value very close to 0, see
test_bug19520.txt.  The trick of the script is the same as your
scenario, with two concurrent workloads:
- One with DROP PROC/CREATE PROC/CALL.
- One with CALL

I had much more success after adding two sleeps to enlarge the
conflict window, see also the sleep.patch attached, for reference.

Finally attached is a patch, where I'd like to propose the
introduction of a path in pgstat_drop_entry() to make the routine able
to accept double drops.

The big comment within pgstat_init_function_usage() documents why it
does its stuff for track_functions, so I was wondering if we should
enforce the same double-drop-acceptance rule for all the callers
everybody, but I also see a point in the correctness, by allowing the
caller to complain if we try to do double drops but error on them,
pointing to a programming error.  Note that
pgstat_drop_entry_internal() is not touched on purpose, to keep the
database-level scans as they are, with double-drops forbidden.

This patch is very close to what Sami has posted on his PGSS thread,
v3-0002, using a missing_ok instead of a skip_dropped:
https://www.postgresql.org/message-id/CAA5RZ0uoxiQ2_=xHGRnyc4WdM9aR0fzdMhBubnw97po==[email-masked]
I didn't suspect that we would need something like that for a
backpatch, but well.

I'm adding Sami in CC in case he wishes to comment on this patch, and
Horiguchi-san as this area of the code concerns him.

Thoughts or comments welcome.
--
Michael
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index d47d05e3d922..eef97a29de7c 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -112,6 +112,7 @@ pgstat_init_function_usage(FunctionCallInfo fcinfo,
 		AcceptInvalidationMessages();
 		if (!SearchSysCacheExists1(PROCOID, ObjectIdGetDatum(fcinfo->flinfo->fn_oid)))
 		{
+			pg_usleep(10000);	/* 10ms */
 			pgstat_drop_entry(PGSTAT_KIND_FUNCTION, MyDatabaseId,
 							  fcinfo->flinfo->fn_oid);
 			ereport(ERROR, errcode(ERRCODE_UNDEFINED_FUNCTION),
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e62979..70a1e302514f 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -85,6 +85,8 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
 			 * Transaction that dropped an object committed. Drop the stats
 			 * too.
 			 */
+			if (it->kind == PGSTAT_KIND_FUNCTION)
+				pg_usleep(500000);	/* 500ms */
 			if (!pgstat_drop_entry(it->kind, it->dboid, objid))
 				not_freed_count++;
 		}
From 29c29dd0a778e952cdf6f55c7be08a33436515e2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email-masked]>
Date: Wed, 17 Jun 2026 12:59:40 +0900
Subject: [PATCH] Fix potential PANICs with concurrent drop of pgstats entries

---
 src/include/utils/pgstat_internal.h             |  3 ++-
 src/backend/utils/activity/pgstat.c             |  2 +-
 src/backend/utils/activity/pgstat_function.c    |  2 +-
 src/backend/utils/activity/pgstat_replslot.c    |  2 +-
 src/backend/utils/activity/pgstat_shmem.c       | 17 ++++++++++++++++-
 src/backend/utils/activity/pgstat_xact.c        |  8 ++++----
 .../test_custom_stats/test_custom_var_stats.c   |  2 +-
 7 files changed, 26 insertions(+), 10 deletions(-)

diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index fe463faaf639..3ca4f4548956 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -807,7 +807,8 @@ extern PgStat_EntryRef *pgstat_get_entry_ref(PgStat_Kind kind, Oid dboid, uint64
 extern bool pgstat_lock_entry(PgStat_EntryRef *entry_ref, bool nowait);
 extern bool pgstat_lock_entry_shared(PgStat_EntryRef *entry_ref, bool nowait);
 extern void pgstat_unlock_entry(PgStat_EntryRef *entry_ref);
-extern bool pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
+extern bool pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid,
+							  bool missing_ok);
 extern void pgstat_drop_all_entries(void);
 extern void pgstat_drop_matching_entries(bool (*do_drop) (PgStatShared_HashEntry *, Datum),
 										 Datum match_data);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index b67da88c7dc2..c4fa14f138fa 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -650,7 +650,7 @@ pgstat_shutdown_hook(int code, Datum arg)
 	dlist_init(&pgStatPending);
 
 	/* drop the backend stats entry */
-	if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber))
+	if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
 		pgstat_request_entry_refs_gc();
 
 	pgstat_detach_shmem();
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index d47d05e3d922..f0366e139907 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -113,7 +113,7 @@ pgstat_init_function_usage(FunctionCallInfo fcinfo,
 		if (!SearchSysCacheExists1(PROCOID, ObjectIdGetDatum(fcinfo->flinfo->fn_oid)))
 		{
 			pgstat_drop_entry(PGSTAT_KIND_FUNCTION, MyDatabaseId,
-							  fcinfo->flinfo->fn_oid);
+							  fcinfo->flinfo->fn_oid, true);
 			ereport(ERROR, errcode(ERRCODE_UNDEFINED_FUNCTION),
 					errmsg("function call to dropped function"));
 		}
diff --git a/src/backend/utils/activity/pgstat_replslot.c b/src/backend/utils/activity/pgstat_replslot.c
index 0d00dd5d93aa..a32b70a03736 100644
--- a/src/backend/utils/activity/pgstat_replslot.c
+++ b/src/backend/utils/activity/pgstat_replslot.c
@@ -188,7 +188,7 @@ pgstat_drop_replslot(ReplicationSlot *slot)
 	Assert(LWLockHeldByMeInMode(ReplicationSlotAllocationLock, LW_EXCLUSIVE));
 
 	if (!pgstat_drop_entry(PGSTAT_KIND_REPLSLOT, InvalidOid,
-						   ReplicationSlotIndex(slot)))
+						   ReplicationSlotIndex(slot), false))
 		pgstat_request_entry_refs_gc();
 }
 
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index b8f354c818a0..ae35332a3347 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -1000,13 +1000,16 @@ pgstat_drop_database_and_contents(Oid dboid)
  * This routine returns false if the stats entry of the dropped object could
  * not be freed, true otherwise.
  *
+ * If missing_ok is true, skip entries that have been concurrently dropped.
+ *
  * The callers of this function should call pgstat_request_entry_refs_gc()
  * if the stats entry could not be freed, to ensure that this entry's memory
  * can be reclaimed later by a different backend calling
  * pgstat_gc_entry_refs().
  */
 bool
-pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid)
+pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid,
+				  bool missing_ok)
 {
 	PgStat_HashKey key = {0};
 	PgStatShared_HashEntry *shent;
@@ -1031,6 +1034,18 @@ pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid)
 	shent = dshash_find(pgStatLocal.shared_hash, &key, true);
 	if (shent)
 	{
+		if (shent->dropped)
+		{
+			if (!missing_ok)
+				elog(ERROR,
+					 "trying to drop stats entry already dropped: kind=%s dboid=%u objid=%" PRIu64,
+					 pgstat_get_kind_info(shent->key.kind)->name,
+					 shent->key.dboid,
+					 shent->key.objid);
+			dshash_release_lock(pgStatLocal.shared_hash, shent);
+			return true;
+		}
+
 		freed = pgstat_drop_entry_internal(shent, NULL);
 
 		/*
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e62979..3e1978775e16 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -85,7 +85,7 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
 			 * Transaction that dropped an object committed. Drop the stats
 			 * too.
 			 */
-			if (!pgstat_drop_entry(it->kind, it->dboid, objid))
+			if (!pgstat_drop_entry(it->kind, it->dboid, objid, true))
 				not_freed_count++;
 		}
 		else if (!isCommit && pending->is_create)
@@ -94,7 +94,7 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
 			 * Transaction that created an object aborted. Drop the stats
 			 * associated with the object.
 			 */
-			if (!pgstat_drop_entry(it->kind, it->dboid, objid))
+			if (!pgstat_drop_entry(it->kind, it->dboid, objid, true))
 				not_freed_count++;
 		}
 
@@ -160,7 +160,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
 			 * Subtransaction creating a new stats object aborted. Drop the
 			 * stats object.
 			 */
-			if (!pgstat_drop_entry(it->kind, it->dboid, objid))
+			if (!pgstat_drop_entry(it->kind, it->dboid, objid, true))
 				not_freed_count++;
 			pfree(pending);
 		}
@@ -323,7 +323,7 @@ pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_item *items,
 		xl_xact_stats_item *it = &items[i];
 		uint64		objid = ((uint64) it->objid_hi) << 32 | it->objid_lo;
 
-		if (!pgstat_drop_entry(it->kind, it->dboid, objid))
+		if (!pgstat_drop_entry(it->kind, it->dboid, objid, true))
 			not_freed_count++;
 	}
 
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 5c4871ed37ca..863d6a524925 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -600,7 +600,7 @@ test_custom_stats_var_drop(PG_FUNCTION_ARGS)
 
 	/* Drop entry and request GC if the entry could not be freed */
 	if (!pgstat_drop_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, InvalidOid,
-						   PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name)))
+						   PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name), false))
 		pgstat_request_entry_refs_gc();
 
 	PG_RETURN_VOID();
-- 
2.54.0

---

From: Ayush Tiwari <[email-masked]>
Date: Wed, 17 Jun 2026 13:49:23 +0530

Hi,

> environments, only my mac is able to get something, because it's

Yeah, you are right, pgss is not a requirement, it just
makes the delay broader.

> introduction of a path in pgstat_drop_entry() to make the routine able

I applied the patch on HEAD and ran my psql harness against it (~60
clients looping DROP / CREATE OR REPLACE / CALL, track_functions=all,
pgss loaded).  Unpatched it PANICs within seconds; with the patch it
stayed up for a ~3 minute run, with the out-of-band drop path firing
several thousand times.  So it clearly closes the hole here.

> Thoughts or comments welcome.

A couple of things which I'm not clear about (these are not blockers
just questions for my understanding):

- With the check moved into the wrapper, pgstat_drop_entry_internal()
  still keeps its own "already dropped" elog().  Every path into
  _internal now seems to guarantee the entry isn't dropped, so
  _internal's copy looks unreachable after the patch
  ,and it's the one with the richer refcount/generation detail.  Was
  the idea to leave it as a backstop, or would folding the handling into
  one place (or making _internal's an Assert) be cleaner?

- In the missing_ok path the wrapper returns true, so the post-commit
  caller skips the not_freed_count++/GC request that a "real" not-freed
  drop would do.  That seems harmless since the entry self-heals
  but was returning true there a deliberate choice over mirroring
  the not-freed/false path? I need to take a look again at this, maybe
  I missed something.

Regards,
Ayush

---

From: Sami Imseih <[email-masked]>
Date: Wed, 17 Jun 2026 15:26:33 -0500

> I didn't suspect that we would need something like that for a

Right, my intention was just adding infrastructure to make tolerating
a dropped entry possible and to be used by an extension in the future.
But, this bug report is timely and now it looks like we need this for
race conditions that are possible in core.

>   ,and it's the one with the richer refcount/generation detail.

Right. Michael's approach of moving the ERROR into the wrapper is better
than keeping it in _internal. Since after the patch no caller enters
pgstat_drop_entry_internal() with a dropped entry
(pgstat_drop_database_and_contents() and pgstat_drop_matching_entries()
already filtered them out, and now the wrapper does too)

> Was the idea to leave it as a backstop, or would folding the handling into

The check in _internal should be converted to an Assert. This documents
that callers  must only pass "live" entries, which will be the case
for all callers after
the patch

>   the not-freed/false path? I need to take a look again at this, maybe

Finding an already dropped entry tells me that the first caller to drop the
entry also triggered a gc request, so we should not request it again.

--
Sami Imseih
Amazon Web Services (AWS)

---

From: Michael Paquier <[email-masked]>
Date: Thu, 18 Jun 2026 10:03:40 +0900

> that callers  must only pass "live" entries, which will be the case

Yeah, I was hesitating to do so, but perhaps you are right that there
is little meaning in keeping this extra elog() anymore in the internal
routine: all its callers discard entries marked as dropped.  And we do
so while holding an exclusive lock.

> entry also triggered a gc request, so we should not request it again.

Nope, we should not trigger multiple requests.

Attaching an updated patch for now.  I am still testing it locally
across all the branches to make sure that the issue is gone (that
takes quite a bit of time).  I'll probably apply it in a few hours
down to v15 if nothing pops up.
--
Michael
From 1addb5f5e70ca6b4b30f79c6b7bd05fbf909bd6b Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email-masked]>
Date: Thu, 18 Jun 2026 09:57:32 +0900
Subject: [PATCH v2] Fix potential PANICs with concurrent drop of pgstats
 entries

---
 src/include/utils/pgstat_internal.h           |  3 ++-
 src/backend/utils/activity/pgstat.c           |  2 +-
 src/backend/utils/activity/pgstat_function.c  |  2 +-
 src/backend/utils/activity/pgstat_replslot.c  |  2 +-
 src/backend/utils/activity/pgstat_shmem.c     | 26 ++++++++++++-------
 src/backend/utils/activity/pgstat_xact.c      |  8 +++---
 .../test_custom_stats/test_custom_var_stats.c |  2 +-
 7 files changed, 27 insertions(+), 18 deletions(-)

 
 	pgstat_detach_shmem();

diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index b8f354c818a0..47c8f927e5b4 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -917,14 +917,7 @@ pgstat_drop_entry_internal(PgStatShared_HashEntry *shent,
 	 * Signal that the entry is dropped - this will eventually cause other
 	 * backends to release their references.
 	 */
-	if (shent->dropped)
-		elog(ERROR,
-			 "trying to drop stats entry already dropped: kind=%s dboid=%u objid=%" PRIu64 " refcount=%u generation=%u",
-			 pgstat_get_kind_info(shent->key.kind)->name,
-			 shent->key.dboid,
-			 shent->key.objid,
-			 pg_atomic_read_u32(&shent->refcount),
-			 pg_atomic_read_u32(&shent->generation));
+	Assert(!shent->dropped);
 	shent->dropped = true;
 
 	/* release refcount marking entry as not dropped */
@@ -1000,13 +993,16 @@ pgstat_drop_database_and_contents(Oid dboid)
@@ -1031,6 +1027,18 @@ pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid)
 
 		/*

 

 	PG_RETURN_VOID();
-- 
2.54.0

---

From: Ayush Tiwari <[email-masked]>
Date: Thu, 18 Jun 2026 08:52:49 +0530

Hi,

> down to v15 if nothing pops up.

The v2 patch looks fine to me. Tested it, no PANICs.

And the assert too looks good to me.

Regards,

---

From: Michael Paquier <[email-masked]>
Date: Thu, 18 Jun 2026 14:40:46 +0900

> The v2 patch looks fine to me. Tested it, no PANICs.

The buildfarm has backfired with the ABI compliance check, which has
made me double-check for some extension code where the API change
would matter, based on this list:
https://wiki.postgresql.org/wiki/CustomCumulativeStats

And I did see one spot here, where there is a
pgstat_custom_drop_entry() that maps to a definition of
pgstat_drop_entry():
https://github.com/pganalyze/pg_stat_plans

I'll go file a ticket.
--
Michael

---

From: Michael Paquier <[email-masked]>
Date: Sat, 20 Jun 2026 16:43:52 +0900

> https://wiki.postgresql.org/wiki/CustomCumulativeStats

Another thing I have forgotten to mention regarding 850b9218c8e4..
I have tweaked the patch so as we still show the refcount and the
generation in the error message, something that was missed in the
latest version of the patch posted on this thread.  This information
is useful for debugging purposes.
--
Michael

---

From: Álvaro Herrera <[email-masked]>
Date: Sat, 20 Jun 2026 13:09:59 +0200

> https://github.com/pganalyze/pg_stat_plans

Is this really the right reaction?  As you know, for the extension
developers it is much more difficult to handle the ABI change on their
side, because they need to force all their users to update the extension
prior to updating Postgres.  This is pretty difficult to do normally and
can lead to crashes on production.  As extension developer you can try
to communicate this, but for end users it is quite easy to miss it.

I think a better answer is to just not introduce the ABI change in
stable branches.  That is, I think we should add a shim function so that
the third-party extensions can continue to use the original ABI; and
only in master you clean that up with a different API, whereby the
extension will be forced to have an #ifdef block for the 19 version or
the older versions, but that's fine because the extension has to be
recompiled for the new major version anyway so the end-user won't be
affected on a minor upgrade.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/

---

From: Michael Paquier <[email-masked]>
Date: Sat, 20 Jun 2026 21:15:50 +0900

> affected on a minor upgrade.

If you feel strongly about it, we could just do something like the
attached in the v15-v18 range.  This introduces a new routine called
pgstat_drop_entry_ext() that gains the new argument "missing_ok", and
pgstat_drop_entry() would be an ABI-compatible wrapper calling it.

What do you think?
--
Michael
From 03e073d2f1d22e777baf62ea2fa8293df76465c4 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email-masked]>
Date: Sat, 20 Jun 2026 21:11:05 +0900
Subject: [PATCH] Re-introduce pgstat_drop_entry(), keeping ABI compatibility

This routine acts as a wrapper of a new pgstat_drop_entry_ext(), used in
the core code with the new missing_ok argument.

Backpatch-through: 15-18
---
 src/include/utils/pgstat_internal.h             |  5 +++--
 src/backend/utils/activity/pgstat.c             |  2 +-
 src/backend/utils/activity/pgstat_function.c    |  4 ++--
 src/backend/utils/activity/pgstat_replslot.c    |  4 ++--
 src/backend/utils/activity/pgstat_shmem.c       | 17 ++++++++++++++---
 src/backend/utils/activity/pgstat_xact.c        |  8 ++++----
 .../modules/injection_points/injection_stats.c  |  4 ++--
 .abi-compliance-history                         |  2 +-
 8 files changed, 29 insertions(+), 17 deletions(-)

diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 4f840c38f072..c09eda3a84d9 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -708,8 +708,9 @@ extern PgStat_EntryRef *pgstat_get_entry_ref(PgStat_Kind kind, Oid dboid, uint64
 extern bool pgstat_lock_entry(PgStat_EntryRef *entry_ref, bool nowait);
 extern bool pgstat_lock_entry_shared(PgStat_EntryRef *entry_ref, bool nowait);
 extern void pgstat_unlock_entry(PgStat_EntryRef *entry_ref);
-extern bool pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid,
-							  bool missing_ok);
+extern bool pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
+extern bool pgstat_drop_entry_ext(PgStat_Kind kind, Oid dboid, uint64 objid,
+								  bool missing_ok);
 extern void pgstat_drop_all_entries(void);
 extern void pgstat_drop_matching_entries(bool (*do_drop) (PgStatShared_HashEntry *, Datum),
 										 Datum match_data);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index d997be2e1d93..99e537bc886a 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -620,7 +620,7 @@ pgstat_shutdown_hook(int code, Datum arg)
 	dlist_init(&pgStatPending);
 
 	/* drop the backend stats entry */
-	if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
+	if (!pgstat_drop_entry_ext(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
 		pgstat_request_entry_refs_gc();
 
 	pgstat_detach_shmem();
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index f763a0d6f5ba..eaeb0dcb1def 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -112,8 +112,8 @@ pgstat_init_function_usage(FunctionCallInfo fcinfo,
 		AcceptInvalidationMessages();
 		if (!SearchSysCacheExists1(PROCOID, ObjectIdGetDatum(fcinfo->flinfo->fn_oid)))
 		{
-			pgstat_drop_entry(PGSTAT_KIND_FUNCTION, MyDatabaseId,
-							  fcinfo->flinfo->fn_oid, true);
+			pgstat_drop_entry_ext(PGSTAT_KIND_FUNCTION, MyDatabaseId,
+								  fcinfo->flinfo->fn_oid, true);
 			ereport(ERROR, errcode(ERRCODE_UNDEFINED_FUNCTION),
 					errmsg("function call to dropped function"));
 		}
diff --git a/src/backend/utils/activity/pgstat_replslot.c b/src/backend/utils/activity/pgstat_replslot.c
index 746b97a0b5af..675c535266c0 100644
--- a/src/backend/utils/activity/pgstat_replslot.c
+++ b/src/backend/utils/activity/pgstat_replslot.c
@@ -157,8 +157,8 @@ pgstat_drop_replslot(ReplicationSlot *slot)
 {
 	Assert(LWLockHeldByMeInMode(ReplicationSlotAllocationLock, LW_EXCLUSIVE));
 
-	if (!pgstat_drop_entry(PGSTAT_KIND_REPLSLOT, InvalidOid,
-						   ReplicationSlotIndex(slot), false))
+	if (!pgstat_drop_entry_ext(PGSTAT_KIND_REPLSLOT, InvalidOid,
+							   ReplicationSlotIndex(slot), false))
 		pgstat_request_entry_refs_gc();
 }
 
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 414b66be2dc4..056c24c6ee1b 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -880,7 +880,7 @@ pgstat_free_entry(PgStatShared_HashEntry *shent, dshash_seq_status *hstat)
 
 /*
  * Helper for both pgstat_drop_database_and_contents() and
- * pgstat_drop_entry(). If hstat is non-null delete the shared entry using
+ * pgstat_drop_entry_ext(). If hstat is non-null delete the shared entry using
  * dshash_delete_current(), otherwise use dshash_delete_entry(). In either
  * case the entry needs to be already locked.
  */
@@ -968,6 +968,17 @@ pgstat_drop_database_and_contents(Oid dboid)
 		pgstat_request_entry_refs_gc();
 }
 
+/*
+ * ABI-preserving wrapper around pgstat_drop_entry_ext().
+ *
+ * The original routine introduced in v15 did not include "missing_ok".
+ */
+bool
+pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid)
+{
+	return pgstat_drop_entry_ext(kind, dboid, objid, false);
+}
+
 /*
  * Drop a single stats entry.
  *
@@ -982,8 +993,8 @@ pgstat_drop_database_and_contents(Oid dboid)
  * pgstat_gc_entry_refs().
  */
 bool
-pgstat_drop_entry(PgStat_Kind kind, Oid dboid, uint64 objid,
-				  bool missing_ok)
+pgstat_drop_entry_ext(PgStat_Kind kind, Oid dboid, uint64 objid,
+					  bool missing_ok)
 {
 	PgStat_HashKey key;
 	PgStatShared_HashEntry *shent;
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index b058c978dcf3..fbbf2844a58f 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -85,7 +85,7 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
 			 * Transaction that dropped an object committed. Drop the stats
 			 * too.
 			 */
-			if (!pgstat_drop_entry(it->kind, it->dboid, objid, true))
+			if (!pgstat_drop_entry_ext(it->kind, it->dboid, objid, true))
 				not_freed_count++;
 		}
 		else if (!isCommit && pending->is_create)
@@ -94,7 +94,7 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
 			 * Transaction that created an object aborted. Drop the stats
 			 * associated with the object.
 			 */
-			if (!pgstat_drop_entry(it->kind, it->dboid, objid, true))
+			if (!pgstat_drop_entry_ext(it->kind, it->dboid, objid, true))
 				not_freed_count++;
 		}
 
@@ -160,7 +160,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
 			 * Subtransaction creating a new stats object aborted. Drop the
 			 * stats object.
 			 */
-			if (!pgstat_drop_entry(it->kind, it->dboid, objid, true))
+			if (!pgstat_drop_entry_ext(it->kind, it->dboid, objid, true))
 				not_freed_count++;
 			pfree(pending);
 		}
@@ -323,7 +323,7 @@ pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_item *items,
 		xl_xact_stats_item *it = &items[i];
 		uint64		objid = ((uint64) it->objid_hi) << 32 | it->objid_lo;
 
-		if (!pgstat_drop_entry(it->kind, it->dboid, objid, true))
+		if (!pgstat_drop_entry_ext(it->kind, it->dboid, objid, true))
 			not_freed_count++;
 	}
 
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
index 435a0484a4ba..c659b2660d4f 100644
--- a/src/test/modules/injection_points/injection_stats.c
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -149,8 +149,8 @@ pgstat_drop_inj(const char *name)
 	if (!inj_stats_loaded || !inj_stats_enabled)
 		return;
 
-	if (!pgstat_drop_entry(PGSTAT_KIND_INJECTION, InvalidOid,
-						   PGSTAT_INJ_IDX(name), false))
+	if (!pgstat_drop_entry_ext(PGSTAT_KIND_INJECTION, InvalidOid,
+							   PGSTAT_INJ_IDX(name), false))
 		pgstat_request_entry_refs_gc();
 }
 
diff --git a/.abi-compliance-history b/.abi-compliance-history
index 6ba3e7519b70..bff1286f9ddd 100644
--- a/.abi-compliance-history
+++ b/.abi-compliance-history
@@ -23,7 +23,7 @@
 # Fix PANIC with track_functions due to concurrent drop of pgstats entries
 # 2026-06-18 11:49:34 +0900
 #
-# This commit has added a "missing_ok" argument to pgstat_drop_entry().  All
+# This commit has added a "missing_ok" argument to pgstat_drop_entry_ext().  All
 # the callers of this routine are in core for v15-v17.  One custom stats kinds
 # available in the public since v18 is impacted (maintainer informed).
 
-- 
2.54.0

---

From: Lukas Fittl <[email-masked]>
Date: Sat, 20 Jun 2026 09:45:05 -0700

> pgstat_drop_entry() would be an ABI-compatible wrapper calling it.

As the developer of the extension in the picture, I was actually
surprised to see the commit making an ABI *and* API breaking change
(and I think with cumulative stats being pluggable in 18, we can
expect people to use public stats functions like pgstat_drop_entry in
extensions), precisely because of the issues Alvaro mentioned.

I think doing it with a new routine is what I would have expected to
happen to preserve API and ABI compatibility, and your follow-up patch
looks good to me.

Thanks,
Lukas

--
Lukas Fittl

---

From: Álvaro Herrera <[email-masked]>
Date: Sat, 20 Jun 2026 18:58:27 +0200

> pgstat_drop_entry() would be an ABI-compatible wrapper calling it.

Yeah, this sounds more or less reasonable.  The callers that pass
missing_ok=false could still use the original function name though, no?

(Personally I would do for an ABI compatibility in back branches with
this new function, and an API breakage in master by simply adding the
new argument everywhere, but keeping the old function name.  This way we
don't preserve unnecessary API ugliness forever.)

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"¿Qué importan los años?  Lo que realmente importa es comprobar que
a fin de cuentas la mejor edad de la vida es estar vivo"  (Mafalda)

---

From: Michael Paquier <[email-masked]>
Date: Sun, 21 Jun 2026 07:51:25 +0900

> don't preserve unnecessary API ugliness forever.)

Yes, that would be the idea:
- On HEAD, keep the old function name, add the parameter.
- On the back-branches, use the new function name with the new
parameter.  And contrary to you limit the use of the old function
name.

Using the old function name in the back-branches where missing_ok is
false would also work, of course.  My suggestion just makes one less
call showing up on the stack.  The previous patch posted is not for
HEAD, only for v15~v18.
--
Michael

---

From: Michael Paquier <[email-masked]>
Date: Mon, 22 Jun 2026 17:28:58 +0900

> call showing up on the stack.  The previous patch posted is not for

By the way, regarding .abi-compliance-history, I am planning to remove
the latest entry after reading how ABICompCheck.pm works in the
buildfarm code.  It uses the latest commit as a base point of
comparison, and compares it with the latest commit specified in the 
ABI file.

Removing the entry in the same commit that adjusts the pgstats routine
to be ABI-compatible should work, and there is no point in adding an
extra entry to re-document the opposite ABI change.  Any comments
perhaps?
--
Michael

---

From: Michael Paquier <[email-masked]>
Date: Tue, 23 Jun 2026 08:12:35 +0900

> comparison, and compares it with the latest commit specified in the 

And adjusted things on v15~v18 as of fe464e9e6863.
--
Michael