agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedFlush some statistics within running transactions
71+ messages / 8 participants
[nested] [flat]
* Flush some statistics within running transactions
@ 2026-01-12 11:03 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-12 11:03 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org
Hi hackers,
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal.
This patch series introduce the ability to $SUBJECT (suggested in [1]) to:
- improve monitoring of long running transactions
- avoid missing places where we should flush statistics (like the one fixed in
039549d70f6)
The patch series is made of 3 sub-patches:
0001: Add pgstat_report_anytime_stat() for periodic stats flushing
It introduces pgstat_report_anytime_stat(), which flushes non transactional
statistics even inside active transactions. A new timeout handler fires every
second to call this function, ensuring timely stats visibility without waiting
for transaction completion.
Implementation details:
- Add PgStat_FlushBehavior enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept
a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY
stats
- Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling
pgstat_report_anytime_stat(false)
Remarks:
- The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate flushing.
The 1 second flush interval is currently hardcoded but we could imagine increase
it or make it configurable. I ran some benchmarks and did not notice any noticeable
performance regression even with a large number of pending entries.
0002: Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to flush some stats. Those calls were in place because
before 0001 stats were flushed only at transaction boundaries.
Remarks:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
- we may want to improve the current behavior to "fix" the 2 above
0003: Add FLUSH_MIXED support and implement it for RELATION stats
This patch extends the non transactional stats infrastructure to support statistics
kinds with mixed transaction behavior: some fields are transactional (e.g., tuple
inserts/updates/deletes) while others are non transactional (e.g., sequential scans
blocks read, ...).
It introduces FLUSH_MIXED as a third flush behavior type, alongside FLUSH_ANYTIME
and FLUSH_AT_TXN_BOUNDARY. For FLUSH_MIXED kinds, a new flush_anytime_cb callback
enables partial flushing of only the non transactional fields during running
transactions.
Some tests are also added.
Implementation details:
- Add FLUSH_MIXED to PgStat_FlushBehavior enum
- Add flush_anytime_cb to PgStat_KindInfo for partial flushing callback
- Update pgstat_flush_pending_entries() to call flush_anytime_cb for
FLUSH_MIXED entries when in anytime_only mode
- Keep FLUSH_MIXED entries in the pending list after partial flush, as
transactional fields still need to be flushed at transaction boundary
RELATION stats are making use of FLUSH_MIXED:
- Change RELATION from TXN_ALL to FLUSH_MIXED
- Implement pgstat_relation_flush_anytime_cb() to flush only read related
stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
blocks_hit
- Clear these fields after flushing to prevent double counting when
pgstat_relation_flush_cb() runs at transaction commit
- Transactional stats (tuples_inserted, tuples_updated, tuples_deleted,
live_tuples, dead_tuples) remain pending until transaction boundary
Remark:
We could also imagine adding a new flush_anytime_static_cb() callback for
future FLUSH_MIXED fixed amount stats.
[1]: https://postgr.es/m/erpzwxoptqhuptdrtehqydzjapvroumkhh7lc6poclbhe7jk7l%40l3yfsq5q4pw7
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v1-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (16.3K, ../../aWTVEycKj7Qh%2FSXH@ip-10-97-1-34.eu-west-3.compute.internal/2-v1-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From 2acc48f3c101b3230090abb53b0e05cc1d8af85f Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v1 1/3] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second to call this function, ensuring timely stats visibility
without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushBehavior enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling
pgstat_report_anytime_stat(false)
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/tcop/postgres.c | 18 +++++
src/backend/utils/activity/pgstat.c | 119 ++++++++++++++++++++++++----
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 15 ++++
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 4 +
src/include/utils/pgstat_internal.h | 11 +++
src/include/utils/timeout.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 154 insertions(+), 17 deletions(-)
9.7% src/backend/tcop/
70.2% src/backend/utils/activity/
9.3% src/backend/utils/init/
6.0% src/include/utils/
4.3% src/include/
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e54bf1e760f..6a91543f80a 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3530,6 +3530,24 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ /* Skip if completely idle */
+ if (!DoingCommandRead || IsTransactionOrTransactionBlock())
+ pgstat_report_anytime_stat(false);
+
+ /* Schedule next timeout */
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
+ PGSTAT_ANYTIME_FLUSH_INTERVAL);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..f7942e47475 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_behavior = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -388,6 +395,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
.shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
@@ -404,6 +412,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter),
.shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
@@ -420,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer),
.shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
@@ -436,6 +446,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +464,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +482,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +788,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1345,9 +1346,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,6 +1383,20 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
@@ -1397,11 +1417,42 @@ pgstat_flush_pending_entries(bool nowait)
cur = next;
}
- Assert(dlist_is_empty(&pgStatPending) == !have_pending);
+ /*
+ * When in anytime_only mode, the list may not be empty because
+ * FLUSH_AT_TXN_BOUNDARY entries were skipped.
+ */
+ Assert(!anytime_only || dlist_is_empty(&pgStatPending) == !have_pending);
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2170,37 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flush non-transactional stats
+ *
+ * This is safe to call even inside a transaction. It only flushes stats
+ * kinds marked as FLUSH_ANYTIME.
+ *
+ * This allows long running transactions to report activity without waiting
+ * for transaction to finish.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /*
+ * Exit if no pending stats at all. This avoids unnecessary work when
+ * backends are idle or in sessions without stats accumulation.
+ *
+ * Note: This check isn't precise as there might be only transactional
+ * stats pending, which we'll skip during the flush. However, maintaining
+ * precise tracking would add complexity that does not seem worth it from
+ * a performance point of view (no noticeable performance regression has
+ * been observed with the current implementation).
+ */
+ if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed)
+ return;
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..cb0f6aecad1 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void);
static void IdleSessionTimeoutHandler(void);
static void IdleStatsUpdateTimeoutHandler(void);
static void ClientCheckTimeoutHandler(void);
+static void AnytimeStatsUpdateTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_ANYTIME_FLUSH_INTERVAL);
}
/*
@@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void)
return result;
}
+
+/*
+ * Timeout handler for flushing non-transactional stats.
+ */
+static void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..8aeb9628871 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..86e65397614 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* When to call pgstat_report_anytime_stat() again */
+#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,6 +536,7 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
extern void pgstat_reset_counters(void);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..02f4f13fc0f 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,14 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush behavior for statistics kinds.
+ */
+typedef enum PgStat_FlushBehavior
+{
+ FLUSH_ANYTIME, /* All fields can flush anytime */
+ FLUSH_AT_TXN_BOUNDARY, /* All fields need transaction boundary */
+} PgStat_FlushBehavior;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +259,9 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /* Flush behavior */
+ PgStat_FlushBehavior flush_behavior;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 09e7f1d420e..9aabb325f16 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2261,6 +2261,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushBehavior
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v1-0002-Remove-useless-calls-to-flush-some-stats.patch (6.4K, ../../aWTVEycKj7Qh%2FSXH@ip-10-97-1-34.eu-west-3.compute.internal/3-v1-0002-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From 2c9e50c45138319660c5aa6860873ffdeebb7a67 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v1 2/3] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
3 files changed, 2 insertions(+), 57 deletions(-)
75.3% src/backend/replication/
24.6% src/backend/utils/activity/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index a41453530a1..266379c780a 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -553,16 +553,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1ab09655a70..c33185bd337 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1826,7 +1822,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1847,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1958,8 +1952,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1968,15 +1961,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2879,8 +2863,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2975,9 +2957,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2985,7 +2964,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2996,21 +2974,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
--
2.34.1
[text/x-diff] v1-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch (15.1K, ../../aWTVEycKj7Qh%2FSXH@ip-10-97-1-34.eu-west-3.compute.internal/4-v1-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch)
download | inline diff:
From af71e6472727b4e18ca369e21e2b4667d4cd172b Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 8 Jan 2026 09:17:38 +0000
Subject: [PATCH v1 3/3] Add FLUSH_MIXED support and implement it for RELATION
stats
This commit extends the non transactional stats infrastructure to support statistics
kinds with mixed transaction behavior: some fields are transactional (e.g., tuple
inserts/updates/deletes) while others are non transactional (e.g., sequential scans
blocks read, ...).
It introduces FLUSH_MIXED as a third flush behavior type, alongside FLUSH_ANYTIME
and FLUSH_AT_TXN_BOUNDARY. For FLUSH_MIXED kinds, a new flush_anytime_cb callback
enables partial flushing of only the non transactional fields during running
transactions.
Some tests are also added.
Implementation details:
- Add FLUSH_MIXED to PgStat_FlushBehavior enum
- Add flush_anytime_cb to PgStat_KindInfo for partial flushing callback
- Update pgstat_flush_pending_entries() to call flush_anytime_cb for
FLUSH_MIXED entries when in anytime_only mode
- Keep FLUSH_MIXED entries in the pending list after partial flush, as
transactional fields still need to be flushed at transaction boundary
RELATION stats are making use of FLUSH_MIXED:
- Change RELATION from TXN_ALL to FLUSH_MIXED
- Implement pgstat_relation_flush_anytime_cb() to flush only read related
stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
blocks_hit
- Clear these fields after flushing to prevent double counting when
pgstat_relation_flush_cb() runs at transaction commit
- Transactional stats (tuples_inserted, tuples_updated, tuples_deleted,
live_tuples, dead_tuples) remain pending until transaction boundary
Remark:
We could also imagine adding a new flush_anytime_static_cb() callback for
future FLUSH_MIXED fixed amount stats.
---
src/backend/utils/activity/pgstat.c | 36 ++++++---
src/backend/utils/activity/pgstat_relation.c | 82 ++++++++++++++++++++
src/include/utils/pgstat_internal.h | 8 ++
src/test/isolation/expected/stats.out | 40 ++++++++++
src/test/isolation/expected/stats_1.out | 40 ++++++++++
src/test/isolation/specs/stats.spec | 12 +++
6 files changed, 209 insertions(+), 9 deletions(-)
56.6% src/backend/utils/activity/
4.7% src/include/utils/
34.0% src/test/isolation/expected/
4.6% src/test/isolation/specs/
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index f7942e47475..191e0ceac88 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -307,7 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
+ .flush_behavior = FLUSH_MIXED,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -315,6 +315,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.pending_size = sizeof(PgStat_TableStatus),
.flush_pending_cb = pgstat_relation_flush_cb,
+ .flush_anytime_cb = pgstat_relation_flush_anytime_cb,
.delete_pending_cb = pgstat_relation_delete_pending_cb,
.reset_timestamp_cb = pgstat_relation_reset_timestamp_cb,
},
@@ -1347,10 +1348,11 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME and FLUSH_MIXED entries,
+ * using flush_anytime_cb for FLUSH_MIXED.
* This is safe to call inside transactions.
*
- * If anytime_only is false, flushes all entries.
+ * If anytime_only is false, flushes all entries using flush_pending_cb.
*/
static bool
pgstat_flush_pending_entries(bool nowait, bool anytime_only)
@@ -1378,6 +1380,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1397,8 +1400,21 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
continue;
}
- /* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ /* flush the stats (with the appropriate callback), if possible */
+ if (anytime_only &&
+ kind_info->flush_behavior == FLUSH_MIXED &&
+ kind_info->flush_anytime_cb != NULL)
+ {
+ /* Partial flush of non-transactional fields only */
+ did_flush = kind_info->flush_anytime_cb(entry_ref, nowait);
+ is_partial_flush = true;
+ }
+ else
+ {
+ /* Full flush */
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ is_partial_flush = false;
+ }
Assert(did_flush || nowait);
@@ -1408,8 +1424,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
@@ -1418,8 +1434,10 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
}
/*
- * When in anytime_only mode, the list may not be empty because
- * FLUSH_AT_TXN_BOUNDARY entries were skipped.
+ * When in anytime_only mode, the list may not be empty even after
+ * successful flushes because FLUSH_AT_TXN_BOUNDARY entries were skipped
+ * or FLUSH_MIXED entries had partial flushes and remain for transaction
+ * boundary.
*/
Assert(!anytime_only || dlist_is_empty(&pgStatPending) == !have_pending);
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index feae2ae5f44..6d6f333039e 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -887,6 +887,88 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+/*
+ * Flush only non-transactional relation stats.
+ *
+ * This is called periodically during running transactions to make some
+ * statistics visible without waiting for the transaction to finish.
+ *
+ * Transactional stats (inserts/updates/deletes and their effects on live/dead
+ * tuple counts) remain in pending until the transaction ends, at which point
+ * pgstat_relation_flush_cb() will flush them.
+ *
+ * If nowait is true and the lock could not be immediately acquired, returns
+ * false without flushing the entry. Otherwise returns true.
+ */
+bool
+pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ Oid dboid;
+ PgStat_TableStatus *lstats; /* pending stats entry */
+ PgStatShared_Relation *shtabstats;
+ PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
+ PgStat_StatDBEntry *dbentry; /* pending database entry */
+ bool has_nontxn_stats = false;
+
+ dboid = entry_ref->shared_entry->key.dboid;
+ lstats = (PgStat_TableStatus *) entry_ref->pending;
+ shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0)
+ has_nontxn_stats = true;
+
+ if (!has_nontxn_stats)
+ return true;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ /* Add only the non-transactional values to the shared entry */
+ tabentry = &shtabstats->stats;
+
+ tabentry->numscans += lstats->counts.numscans;
+ if (lstats->counts.numscans)
+ {
+ TimestampTz t = GetCurrentTimestamp();
+
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+ tabentry->tuples_returned += lstats->counts.tuples_returned;
+ tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+ tabentry->blocks_fetched += lstats->counts.blocks_fetched;
+ tabentry->blocks_hit += lstats->counts.blocks_hit;
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ dbentry->tuples_returned += lstats->counts.tuples_returned;
+ dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ dbentry->blocks_fetched += lstats->counts.blocks_fetched;
+ dbentry->blocks_hit += lstats->counts.blocks_hit;
+
+ /*
+ * Clear the flushed fields from pending stats to prevent double-counting
+ * when pgstat_relation_flush_cb() runs at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
+}
+
void
pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref)
{
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 02f4f13fc0f..85d92f4c945 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -231,6 +231,7 @@ typedef enum PgStat_FlushBehavior
{
FLUSH_ANYTIME, /* All fields can flush anytime */
FLUSH_AT_TXN_BOUNDARY, /* All fields need transaction boundary */
+ FLUSH_MIXED, /* MIXED so needs callbacks */
} PgStat_FlushBehavior;
/*
@@ -262,6 +263,12 @@ typedef struct PgStat_KindInfo
/* Flush behavior */
PgStat_FlushBehavior flush_behavior;
+ /*
+ * For PGSTAT_FLUSH_MIXED kinds: callback to flush only some fields. If
+ * NULL for a MIXED kind, treated as PGSTAT_FLUSH_AT_TXN_BOUNDARY.
+ */
+ bool (*flush_anytime_cb) (PgStat_EntryRef *entry_ref, bool nowait);
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -774,6 +781,7 @@ extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..6d62b30e4a7 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..2fade10e817 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 0| 0| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..1b0168e6176 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than PGSTAT_ANYTIME_FLUSH_INTERVAL
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -138,6 +140,7 @@ step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -435,6 +438,15 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-15 03:54 ` Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-01-15 03:54 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org
Hi,
Thanks for these patches!
I took a quick look at the patches and I have some general comments.
> Long running transactions can accumulate significant statistics (WAL, IO, ...)
> that remain unflushed until the transaction ends. This delays visibility of
> resource usage in monitoring views like pg_stat_io and pg_stat_wal.
+1. I do think this is a good idea. Long-running transactions cause accumulated
stats to appear as spikes in monitoring tools rather than as gradual activity.
This would help level out, though not eliminate, those artificial spikes.
> The 1 second flush interval is currently hardcoded but we could imagine increase
> it or make it configurable.
Someone may want to turn this off as well. I think a GUC will be needed.
> RELATION stats are making use of FLUSH_MIXED:
> stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
> blocks_hit
I’m concerned that fields being temporarily out of sync might impact monitoring
calculations, if the formula is dealing with fields that have
different flush strategies.
That said, minor discrepancies are usually tolerable for monitoring
data analysis.
For the numscans, should we not also update the scan timestamp?
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-15 12:18 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-15 12:18 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org
Hi,
On Wed, Jan 14, 2026 at 09:54:17PM -0600, Sami Imseih wrote:
> I took a quick look at the patches and I have some general comments.
Thanks!
>
> > Long running transactions can accumulate significant statistics (WAL, IO, ...)
> > that remain unflushed until the transaction ends. This delays visibility of
> > resource usage in monitoring views like pg_stat_io and pg_stat_wal.
>
> +1. I do think this is a good idea. Long-running transactions cause accumulated
> stats to appear as spikes in monitoring tools rather than as gradual activity.
> This would help level out, though not eliminate, those artificial spikes.
Yeah.
> > The 1 second flush interval is currently hardcoded but we could imagine increase
> > it or make it configurable.
>
> Someone may want to turn this off as well. I think a GUC will be needed.
I gave this more thoughts and I wonder if this should be configurable at all.
I mean, we don't do it for PGSTAT_MIN_INTERVAL, PGSTAT_MAX_INTERVAL and
PGSTAT_IDLE_INTERVAL. We could imagine make it configurable if it produces
noticeable performance impact but that's not what I observed.
> > RELATION stats are making use of FLUSH_MIXED:
>
> > stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
> > blocks_hit
>
> I’m concerned that fields being temporarily out of sync might impact monitoring
> calculations, if the formula is dealing with fields that have
> different flush strategies.
That's a good point. Maybe we should document the fields flush strategy?
> That said, minor discrepancies are usually tolerable for monitoring
> data analysis.
>
> For the numscans, should we not also update the scan timestamp?
The problem is that we could not call GetCurrentTransactionStopTimestamp(), so
we would need to call GetCurrentTimestamp() instead. I'm not sure that calling
GetCurrentTimestamp() every second would be a real issue though, and if it is
maybe we could increase this 1s value.
That said I agree that having seq_scan being updated and not last_seq_scan is not
that great.
Maybe we should keep this in mind and see what to do depending where this thread
is going (I mean if the current proposed design has to be changed).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-15 17:25 ` Sami Imseih <samimseih@gmail.com>
2026-01-15 18:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 2 replies; 71+ messages in thread
From: Sami Imseih @ 2026-01-15 17:25 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org
> > > The 1 second flush interval is currently hardcoded but we could imagine increase
> > > it or make it configurable.
> >
> > Someone may want to turn this off as well. I think a GUC will be needed.
>
> I gave this more thoughts and I wonder if this should be configurable at all.
> I mean, we don't do it for PGSTAT_MIN_INTERVAL, PGSTAT_MAX_INTERVAL and
> PGSTAT_IDLE_INTERVAL. We could imagine make it configurable if it produces
> noticeable performance impact but that's not what I observed.
Is there a reason we need a new constant (PGSTAT_ANYTIME_FLUSH_INTERVAL)
for anytime flushes and can't rely on the existing PGSTAT_MIN_INTERVAL?
Also, How did you benchmark? I am less concerned about long running
transactions,
background processes and more about short/high concurrency transactions seeing
additional overhead due to additional flushing. Is that latter a concern?
> > > stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
> > > blocks_hit
> >
> > I’m concerned that fields being temporarily out of sync might impact monitoring
> > calculations, if the formula is dealing with fields that have
> > different flush strategies.
>
> That's a good point. Maybe we should document the fields flush strategy?
Yeah, we will need to document this.
> > That said, minor discrepancies are usually tolerable for monitoring
> > data analysis.
> >
> > For the numscans, should we not also update the scan timestamp?
>
> The problem is that we could not call GetCurrentTransactionStopTimestamp(), so
> we would need to call GetCurrentTimestamp() instead. I'm not sure that calling
> GetCurrentTimestamp() every second would be a real issue though, and if it is
> maybe we could increase this 1s value.
> That said I agree that having seq_scan being updated and not last_seq_scan is not
> that great.
with v3 , I checked by running seq scans in a long running transaction,
and I observed both for these values being updated at the same time. I think
this is OK.
# pgstat_relation_flush_anytime_cb
```
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
TimestampTz t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
}
```
and
# pgstat_relation_flush_cb
```
if (lstats->counts.numscans)
{
TimestampTz t = GetCurrentTransactionStopTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
}
```
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-15 18:16 ` Sami Imseih <samimseih@gmail.com>
1 sibling, 0 replies; 71+ messages in thread
From: Sami Imseih @ 2026-01-15 18:16 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org
> with v3 , I checked by running seq scans in a long running transaction,
Sorry I mean 0003
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-16 11:37 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-16 11:37 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org
Hi,
On Thu, Jan 15, 2026 at 11:25:18AM -0600, Sami Imseih wrote:
> > > > The 1 second flush interval is currently hardcoded but we could imagine increase
> > > > it or make it configurable.
> > >
> > > Someone may want to turn this off as well. I think a GUC will be needed.
> >
> > I gave this more thoughts and I wonder if this should be configurable at all.
> > I mean, we don't do it for PGSTAT_MIN_INTERVAL, PGSTAT_MAX_INTERVAL and
> > PGSTAT_IDLE_INTERVAL. We could imagine make it configurable if it produces
> > noticeable performance impact but that's not what I observed.
>
> Is there a reason we need a new constant (PGSTAT_ANYTIME_FLUSH_INTERVAL)
> for anytime flushes and can't rely on the existing PGSTAT_MIN_INTERVAL?
It currently gives flexibility for testing. If we agree that 1s is the right value
to set and that it should not be configurable then yeah we could replace it with
PGSTAT_MIN_INTERVAL then.
> Also, How did you benchmark? I am less concerned about long running
> transactions,
> background processes and more about short/high concurrency transactions seeing
> additional overhead due to additional flushing. Is that latter a concern?
I ran 3 kinds of tests:
1/
pgbench -c 32 -j 4 -T 60 -f short.sql -n -r $DB
with short.sql:
\set t1 random(1, 100)
\set t2 random(1, 100)
\set t3 random(1, 100)
\set t4 random(1, 100)
\set t5 random(1, 100)
\set t6 random(1, 100)
\set t7 random(1, 100)
\set t8 random(1, 100)
\set t9 random(1, 100)
\set t10 random(1, 100)
\set row random(1, 1000)
BEGIN;
UPDATE t:t1 SET val = val + 1 WHERE id = :row;
UPDATE t:t2 SET val = val + 1 WHERE id = :row;
UPDATE t:t3 SET val = val + 1 WHERE id = :row;
UPDATE t:t4 SET val = val + 1 WHERE id = :row;
UPDATE t:t5 SET val = val + 1 WHERE id = :row;
UPDATE t:t6 SET val = val + 1 WHERE id = :row;
UPDATE t:t7 SET val = val + 1 WHERE id = :row;
UPDATE t:t8 SET val = val + 1 WHERE id = :row;
UPDATE t:t9 SET val = val + 1 WHERE id = :row;
UPDATE t:t10 SET val = val + 1 WHERE id = :row;
COMMIT;
2/
psql $DB -f long.sql
with long.sql:
DO $$
BEGIN
FOR i IN 1..100 LOOP
EXECUTE format('TRUNCATE TABLE t%s', i);
EXECUTE format('INSERT INTO t%s SELECT generate_series(1, 1000000)', i);
EXECUTE format('UPDATE t%s SET val = val + 1', i);
EXECUTE format('SELECT COUNT(1) FROM t%s', i);
END LOOP;
END $$;
3/
pgbench -i -s 50 $DB
pgbench -c 32 -j 4 -T 60 -N -n -r $DB
I don't think this feature could add a noticeable performance impact, so the tests
have been that simple. Do you think we should worry more?
> > > I’m concerned that fields being temporarily out of sync might impact monitoring
> > > calculations, if the formula is dealing with fields that have
> > > different flush strategies.
> >
> > That's a good point. Maybe we should document the fields flush strategy?
>
> Yeah, we will need to document this.
Will do in the next version.
> I checked by running seq scans in a long running transaction,
> and I observed both for these values being updated at the same time. I think
> this is OK.
>
I do think the same.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-16 16:44 ` Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-01-16 16:44 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org
I took a look at 0001 in depth.
> I don't think this feature could add a noticeable performance impact, so the tests
> have been that simple. Do you think we should worry more?
One observation is there's no coordination between ANYTIME and
TXN_BOUNDARY flushes. While PGSTAT_MIN_INTERVAL
prevents a backend from flushing more than once per second, a backend can
still perform both an ANYTIME flush and a TXN_BOUNDARY flush within
the same 1-second window. Not saying this will be a real problem in
the real-world,
but we definitely took measures in the current implementation to avoid
this scenario.
A few other comments on 0001
+ /* Skip if completely idle */
+ if (!DoingCommandRead || IsTransactionOrTransactionBlock())
+ pgstat_report_anytime_stat(false);
Does this need to be conditional? worst case, we return right away with an empty
list. Best case, is we are consistently flushing.
+ /*
+ * When in anytime_only mode, the list may not be empty because
+ * FLUSH_AT_TXN_BOUNDARY entries were skipped.
+ */
+ Assert(!anytime_only || dlist_is_empty(&pgStatPending) ==
!have_pending);
Checking for !anytime_only is unnecessary here.
"list_is_empty(&pgStatPending) == !have_pending"
should be true regardless of ANYTIME or TXN_BOUNDARY, right?
Below are a couple of edits for comments I felt would improve
readability of the code.
1/
/*
- * Flush non-transactional stats
- *
- * This is safe to call even inside a transaction. It only flushes stats
- * kinds marked as FLUSH_ANYTIME.
- *
- * This allows long running transactions to report activity without waiting
- * for transaction to finish.
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
*/
2/
typedef enum PgStat_FlushBehavior
{
- FLUSH_ANYTIME, /* All fields can
flush anytime */
- FLUSH_AT_TXN_BOUNDARY, /* All fields need transaction
boundary */
+ FLUSH_ANYTIME, /* All fields can be
flushed anytime,
+ *
including within transactions */
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ *
transaction boundary */
} PgStat_FlushBehavior;
I will start looking at the remaining patches next.
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-19 12:17 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-20 23:18 ` Re: Flush some statistics within running transactions Zsolt Parragi <zsolt.parragi@percona.com>
0 siblings, 2 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-19 12:17 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org
Hi,
On Fri, Jan 16, 2026 at 10:44:48AM -0600, Sami Imseih wrote:
> I took a look at 0001 in depth.
Thanks!
> > I don't think this feature could add a noticeable performance impact, so the tests
> > have been that simple. Do you think we should worry more?
>
> One observation is there's no coordination between ANYTIME and
> TXN_BOUNDARY flushes. While PGSTAT_MIN_INTERVAL
> prevents a backend from flushing more than once per second, a backend can
> still perform both an ANYTIME flush and a TXN_BOUNDARY flush within
> the same 1-second window. Not saying this will be a real problem in
> the real-world,
> but we definitely took measures in the current implementation to avoid
> this scenario.
Right. I think that the PGSTAT_MIN_INTERVAL throttling was put in place to prevent
flushing too frequently when the backend has a high commit rate. But here, while
it's true that we don't follow that rule (means a backend could flush more than one
time per second), that would be a maximum of 2 times (given that ANYTIME is
flushing every second). So, I'm not sure that this single extra flush is worth
worrying about. Plus we'd certainly need an extra GetCurrentTimestamp() call, so
I'm not sure it's worth it.
> A few other comments on 0001
>
> + /* Skip if completely idle */
> + if (!DoingCommandRead || IsTransactionOrTransactionBlock())
> + pgstat_report_anytime_stat(false);
>
> Does this need to be conditional? worst case, we return right away with an empty
> list. Best case, is we are consistently flushing.
Yeah, I think we could remove this check and just rely on the ones in
pgstat_report_anytime_stat(). Done in the attached.
> + Assert(!anytime_only || dlist_is_empty(&pgStatPending) ==
> !have_pending);
>
> Checking for !anytime_only is unnecessary here.
> "list_is_empty(&pgStatPending) == !have_pending"
> should be true regardless of ANYTIME or TXN_BOUNDARY, right?
Right, thanks for catching it, it was remaining garbage from my dev iterations.
> Below are a couple of edits for comments I felt would improve
> readability of the code.
Done as suggested.
> I will start looking at the remaining patches next.
Thanks!
Note that I also updated the doc in 0003 for the stats that have mixed fields.
BTW, I think that we could also make the Function stat kind as flush any time,
thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v2-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (16.2K, ../../aW4gwqVybB4INCvu@ip-10-97-1-34.eu-west-3.compute.internal/2-v2-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From 605cae0291397047b09aa025b742bdcaf9bdd528 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v2 1/3] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second to call this function, ensuring timely stats visibility
without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushBehavior enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling
pgstat_report_anytime_stat(false)
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/tcop/postgres.c | 16 ++++
src/backend/utils/activity/pgstat.c | 113 ++++++++++++++++++++++++----
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 15 ++++
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 4 +
src/include/utils/pgstat_internal.h | 13 ++++
src/include/utils/timeout.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 149 insertions(+), 16 deletions(-)
8.2% src/backend/tcop/
69.3% src/backend/utils/activity/
9.7% src/backend/utils/init/
7.7% src/include/utils/
4.5% src/include/
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e54bf1e760f..9c4a9078ee0 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3530,6 +3530,22 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+
+ /* Schedule next timeout */
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
+ PGSTAT_ANYTIME_FLUSH_INTERVAL);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..0f45a7d165e 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_behavior = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -388,6 +395,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
.shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
@@ -404,6 +412,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter),
.shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
@@ -420,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer),
.shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
@@ -436,6 +446,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +464,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +482,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_behavior = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +788,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1345,9 +1346,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,6 +1383,20 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
@@ -1397,11 +1417,42 @@ pgstat_flush_pending_entries(bool nowait)
cur = next;
}
+ /*
+ * When in anytime_only mode, the list may not be empty because
+ * FLUSH_AT_TXN_BOUNDARY entries were skipped.
+ */
Assert(dlist_is_empty(&pgStatPending) == !have_pending);
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /*
+ * Exit if no pending stats at all. This avoids unnecessary work when
+ * backends are idle or in sessions without stats accumulation.
+ *
+ * Note: This check isn't precise as there might be only transactional
+ * stats pending, which we'll skip during the flush. However, maintaining
+ * precise tracking would add complexity that does not seem worth it from
+ * a performance point of view (no noticeable performance regression has
+ * been observed with the current implementation).
+ */
+ if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed)
+ return;
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..cb0f6aecad1 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void);
static void IdleSessionTimeoutHandler(void);
static void IdleStatsUpdateTimeoutHandler(void);
static void ClientCheckTimeoutHandler(void);
+static void AnytimeStatsUpdateTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_ANYTIME_FLUSH_INTERVAL);
}
/*
@@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void)
return result;
}
+
+/*
+ * Timeout handler for flushing non-transactional stats.
+ */
+static void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..8aeb9628871 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..86e65397614 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* When to call pgstat_report_anytime_stat() again */
+#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,6 +536,7 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
extern void pgstat_reset_counters(void);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..63feae640d1 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,16 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush behavior for statistics kinds.
+ */
+typedef enum PgStat_FlushBehavior
+{
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+} PgStat_FlushBehavior;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +261,9 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /* Flush behavior */
+ PgStat_FlushBehavior flush_behavior;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3f3a888fd0e..610b35a9b31 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushBehavior
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v2-0002-Remove-useless-calls-to-flush-some-stats.patch (6.4K, ../../aW4gwqVybB4INCvu@ip-10-97-1-34.eu-west-3.compute.internal/3-v2-0002-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From e012b211f86cba606375c8730f12a4d25dae93d4 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v2 2/3] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
3 files changed, 2 insertions(+), 57 deletions(-)
75.3% src/backend/replication/
24.6% src/backend/utils/activity/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index a41453530a1..266379c780a 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -553,16 +553,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1ab09655a70..c33185bd337 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1826,7 +1822,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1847,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1958,8 +1952,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1968,15 +1961,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2879,8 +2863,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2975,9 +2957,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2985,7 +2964,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2996,21 +2974,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
--
2.34.1
[text/x-diff] v2-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch (17.5K, ../../aW4gwqVybB4INCvu@ip-10-97-1-34.eu-west-3.compute.internal/4-v2-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch)
download | inline diff:
From d71d13b8e5a938a8a94362121fa937f9026fb51a Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v2 3/3] Add FLUSH_MIXED support and implement it for RELATION
stats
This commit extends the non transactional stats infrastructure to support statistics
kinds with mixed transaction behavior: some fields are transactional (e.g., tuple
inserts/updates/deletes) while others are non transactional (e.g., sequential scans
blocks read, ...).
It introduces FLUSH_MIXED as a third flush behavior type, alongside FLUSH_ANYTIME
and FLUSH_AT_TXN_BOUNDARY. For FLUSH_MIXED kinds, a new flush_anytime_cb callback
enables partial flushing of only the non transactional fields during running
transactions.
Some tests are also added.
Implementation details:
- Add FLUSH_MIXED to PgStat_FlushBehavior enum
- Add flush_anytime_cb to PgStat_KindInfo for partial flushing callback
- Update pgstat_flush_pending_entries() to call flush_anytime_cb for
FLUSH_MIXED entries when in anytime_only mode
- Keep FLUSH_MIXED entries in the pending list after partial flush, as
transactional fields still need to be flushed at transaction boundary
RELATION stats are making use of FLUSH_MIXED:
- Change RELATION from TXN_ALL to FLUSH_MIXED
- Implement pgstat_relation_flush_anytime_cb() to flush only read related
stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
blocks_hit
- Clear these fields after flushing to prevent double counting when
pgstat_relation_flush_cb() runs at transaction commit
- Transactional stats (tuples_inserted, tuples_updated, tuples_deleted,
live_tuples, dead_tuples) remain pending until transaction boundary
Remark:
We could also imagine adding a new flush_anytime_static_cb() callback for
future FLUSH_MIXED fixed amount stats.
---
doc/src/sgml/monitoring.sgml | 30 +++++++
src/backend/utils/activity/pgstat.c | 36 ++++++---
src/backend/utils/activity/pgstat_relation.c | 82 ++++++++++++++++++++
src/include/utils/pgstat_internal.h | 9 +++
src/test/isolation/expected/stats.out | 40 ++++++++++
src/test/isolation/expected/stats_1.out | 40 ++++++++++
src/test/isolation/specs/stats.spec | 12 +++
7 files changed, 237 insertions(+), 12 deletions(-)
14.5% doc/src/sgml/
47.2% src/backend/utils/activity/
4.7% src/include/utils/
29.4% src/test/isolation/expected/
4.0% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 817fd9f4ca7..15b55016b66 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3730,6 +3730,16 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ All the statistics are updated while the transactions are in progress, except
+ for <structfield>xact_commit</structfield>, <structfield>xact_rollback</structfield>,
+ <structfield>tup_inserted</structfield>, <structfield>tup_updated</structfield> and
+ <structfield>tup_deleted</structfield> that are updated only when the transactions
+ finish.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -4186,6 +4196,17 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress. This means that we can see
+ those statistics being updated without having to wait until the transaction
+ finishes.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4367,6 +4388,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress. This means that we can see
+ those statistics being updated without having to wait until the transaction
+ finishes.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 0f45a7d165e..5b93683ea9b 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -289,7 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
+ .flush_behavior = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -307,7 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_behavior = FLUSH_AT_TXN_BOUNDARY,
+ .flush_behavior = FLUSH_MIXED,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -315,6 +315,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.pending_size = sizeof(PgStat_TableStatus),
.flush_pending_cb = pgstat_relation_flush_cb,
+ .flush_anytime_cb = pgstat_relation_flush_anytime_cb,
.delete_pending_cb = pgstat_relation_delete_pending_cb,
.reset_timestamp_cb = pgstat_relation_reset_timestamp_cb,
},
@@ -1347,10 +1348,11 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME and FLUSH_MIXED entries,
+ * using flush_anytime_cb for FLUSH_MIXED.
* This is safe to call inside transactions.
*
- * If anytime_only is false, flushes all entries.
+ * If anytime_only is false, flushes all entries using flush_pending_cb.
*/
static bool
pgstat_flush_pending_entries(bool nowait, bool anytime_only)
@@ -1378,6 +1380,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1397,8 +1400,21 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
continue;
}
- /* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ /* flush the stats (with the appropriate callback), if possible */
+ if (anytime_only &&
+ kind_info->flush_behavior == FLUSH_MIXED &&
+ kind_info->flush_anytime_cb != NULL)
+ {
+ /* Partial flush of non-transactional fields only */
+ did_flush = kind_info->flush_anytime_cb(entry_ref, nowait);
+ is_partial_flush = true;
+ }
+ else
+ {
+ /* Full flush */
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ is_partial_flush = false;
+ }
Assert(did_flush || nowait);
@@ -1408,8 +1424,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
@@ -1417,10 +1433,6 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
cur = next;
}
- /*
- * When in anytime_only mode, the list may not be empty because
- * FLUSH_AT_TXN_BOUNDARY entries were skipped.
- */
Assert(dlist_is_empty(&pgStatPending) == !have_pending);
return have_pending;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index feae2ae5f44..6d6f333039e 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -887,6 +887,88 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+/*
+ * Flush only non-transactional relation stats.
+ *
+ * This is called periodically during running transactions to make some
+ * statistics visible without waiting for the transaction to finish.
+ *
+ * Transactional stats (inserts/updates/deletes and their effects on live/dead
+ * tuple counts) remain in pending until the transaction ends, at which point
+ * pgstat_relation_flush_cb() will flush them.
+ *
+ * If nowait is true and the lock could not be immediately acquired, returns
+ * false without flushing the entry. Otherwise returns true.
+ */
+bool
+pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ Oid dboid;
+ PgStat_TableStatus *lstats; /* pending stats entry */
+ PgStatShared_Relation *shtabstats;
+ PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
+ PgStat_StatDBEntry *dbentry; /* pending database entry */
+ bool has_nontxn_stats = false;
+
+ dboid = entry_ref->shared_entry->key.dboid;
+ lstats = (PgStat_TableStatus *) entry_ref->pending;
+ shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0)
+ has_nontxn_stats = true;
+
+ if (!has_nontxn_stats)
+ return true;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ /* Add only the non-transactional values to the shared entry */
+ tabentry = &shtabstats->stats;
+
+ tabentry->numscans += lstats->counts.numscans;
+ if (lstats->counts.numscans)
+ {
+ TimestampTz t = GetCurrentTimestamp();
+
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+ tabentry->tuples_returned += lstats->counts.tuples_returned;
+ tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+ tabentry->blocks_fetched += lstats->counts.blocks_fetched;
+ tabentry->blocks_hit += lstats->counts.blocks_hit;
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ dbentry->tuples_returned += lstats->counts.tuples_returned;
+ dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ dbentry->blocks_fetched += lstats->counts.blocks_fetched;
+ dbentry->blocks_hit += lstats->counts.blocks_hit;
+
+ /*
+ * Clear the flushed fields from pending stats to prevent double-counting
+ * when pgstat_relation_flush_cb() runs at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
+}
+
void
pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref)
{
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 63feae640d1..c80b8162b37 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -233,6 +233,8 @@ typedef enum PgStat_FlushBehavior
* including within transactions */
FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
* transaction boundary */
+ FLUSH_MIXED, /* Mix of fields that can be flushed anytime
+ * or only at transaction boundary */
} PgStat_FlushBehavior;
/*
@@ -264,6 +266,12 @@ typedef struct PgStat_KindInfo
/* Flush behavior */
PgStat_FlushBehavior flush_behavior;
+ /*
+ * For PGSTAT_FLUSH_MIXED kinds: callback to flush only some fields. If
+ * NULL for a MIXED kind, treated as PGSTAT_FLUSH_AT_TXN_BOUNDARY.
+ */
+ bool (*flush_anytime_cb) (PgStat_EntryRef *entry_ref, bool nowait);
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -776,6 +784,7 @@ extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..6d62b30e4a7 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..2fade10e817 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 0| 0| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..1b0168e6176 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than PGSTAT_ANYTIME_FLUSH_INTERVAL
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -138,6 +140,7 @@ step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -435,6 +438,15 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-20 19:27 ` Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-01-20 19:27 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org
Thanks for the updates!
> > > I don't think this feature could add a noticeable performance impact, so the tests
> > > have been that simple. Do you think we should worry more?
> >
> > One observation is there's no coordination between ANYTIME and
> > TXN_BOUNDARY flushes. While PGSTAT_MIN_INTERVAL
> > prevents a backend from flushing more than once per second, a backend can
> > still perform both an ANYTIME flush and a TXN_BOUNDARY flush within
> > the same 1-second window. Not saying this will be a real problem in
> > the real-world,
> > but we definitely took measures in the current implementation to avoid
> > this scenario.
>
> Right. I think that the PGSTAT_MIN_INTERVAL throttling was put in place to prevent
> flushing too frequently when the backend has a high commit rate. But here, while
> it's true that we don't follow that rule (means a backend could flush more than one
> time per second), that would be a maximum of 2 times (given that ANYTIME is
> flushing every second). So, I'm not sure that this single extra flush is worth
> worrying about. Plus we'd certainly need an extra GetCurrentTimestamp() call, so
> I'm not sure it's worth it.
Yeah, all PGSTAT_MIN_INTERVAL does is throttle pgstat_flush_pending_entries.
Even in the current state, it does not limit how many kinds are flushed, etc.
I consider the ANYTIME flushes the same as just adding another stats kind.
So, I am not really worried about either.
I have some more comments:
-- v2-0001
#1.
+/* When to call pgstat_report_anytime_stat() again */
+#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000
+
We should just use PGSTAT_MIN_INTERVAL.
#2.
instead of ".flush_behavior", maybe ".flush_mode"? "mode" in the name is better
for configuration fields.
#3.
+/*
+ * Flush behavior for statistics kinds.
+ */
+typedef enum PgStat_FlushBehavior
+{
+ FLUSH_ANYTIME, /* All fields can be
flushed anytime,
+ *
including within transactions */
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ *
transaction boundary */
+} PgStat_FlushBehavior;
FLUSH_AT_TXN_BOUNDARY should be the first value in PgStat_FlushBehavior.
Otherwise kinds ( built-in or custom ) that do not specify a flush_behavior
will default to FLUSH_ANYTIME. I don't think this is what we want.
FLUSH_AT_TXN_BOUNDARY should be the default.
#4. Can we add a test here? Maybe generate some wal inside a long
running transaction and
make sure the stats are updated after > 1 second
-- v2-0002
No comments for this one. With ANYTIME, indeed those flushes are not needed.
-- v2-0003
#1. Should we maybe make this a bit longer? maybe 2 or 3 seconds?
May make the tests slightly longer, but maybe better for test stability.
```
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
```
#2.
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0)
+ has_nontxn_stats = true;
+
+ if (!has_nontxn_stats)
+ return true;
Can we just do this without a has_nontxn_stats?
This is also the same patter as a regular flush, although
in the case `pg_memory_is_all_zeros` is used.
```
if (lstats->counts.numscans == 0 &&
lstats->counts.tuples_returned == 0 &&
lstats->counts.tuples_fetched == 0 &&
lstats->counts.blocks_fetched == 0 &&
lstats->counts.blocks_hit == 0)
return true;
```
#3.
+ are updated while the transactions are in progress. This means
that we can see
+ those statistics being updated without having to wait until the transaction
+ finishes.
+ </para>
The "This means ...... " line used several times does not add value, IMO.
"are updated while the transactions are in progress." is sufficient.
#4.
+ <note>
+ <para>
+ All the statistics are updated while the transactions are in
progress, except
+ for <structfield>xact_commit</structfield>,
<structfield>xact_rollback</structfield>,
+ <structfield>tup_inserted</structfield>,
<structfield>tup_updated</structfield> and
+ <structfield>tup_deleted</structfield> that are updated only when
the transactions
+ finish.
+ </para>
+ </note>
Only these 5 fields from pgstat_relation_flush_anytime_cb, so only the below are
"All the statistics are updated while the transactions are in progress", right?
numscans
tuples_returned
tuples_fetched
blocks_fetched
blocks_hit
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-21 10:34 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-21 23:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-21 10:34 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Tue, Jan 20, 2026 at 01:27:55PM -0600, Sami Imseih wrote:
> I have some more comments:
Thanks!
> -- v2-0001
>
> #1.
>
> +/* When to call pgstat_report_anytime_stat() again */
> +#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000
> +
>
> We should just use PGSTAT_MIN_INTERVAL.
Okay, done. We can still switch to a dedicated one if we feel the need later on.
> #2.
>
> instead of ".flush_behavior", maybe ".flush_mode"? "mode" in the name is better
> for configuration fields.
Sounds good.
> #3.
>
> FLUSH_AT_TXN_BOUNDARY should be the first value in PgStat_FlushBehavior.
> Otherwise kinds ( built-in or custom ) that do not specify a flush_behavior
> will default to FLUSH_ANYTIME. I don't think this is what we want.
> FLUSH_AT_TXN_BOUNDARY should be the default.
Good point, agreed and done.
> #4. Can we add a test here? Maybe generate some wal inside a long
> running transaction and
> make sure the stats are updated after > 1 second
I'm not sure, that's also somehow the purpose of 0002 (with 039549d70f6 being
reverted).
0001 and 0002 could be merged and pushed as one commit. That said I'm not opposed
if you feel strongly about it.
> -- v2-0003
>
> #1. Should we maybe make this a bit longer? maybe 2 or 3 seconds?
> May make the tests slightly longer, but maybe better for test stability.
>
> ```
> +step s1_sleep: SELECT pg_sleep(1.5);
> +pg_sleep
> +--------
> ```
Not sure, we could increase if we see the test failing.
> #2.
> + /*
> + * Check if there are any non-transactional stats to flush. Avoid
> + * unnecessarily locking the entry if nothing accumulated.
> + */
> + if (lstats->counts.numscans > 0 ||
> + lstats->counts.tuples_returned > 0 ||
> + lstats->counts.tuples_fetched > 0 ||
> + lstats->counts.blocks_fetched > 0 ||
> + lstats->counts.blocks_hit > 0)
> + has_nontxn_stats = true;
> +
> + if (!has_nontxn_stats)
> + return true;
> Can we just do this without a has_nontxn_stats?
Yeah.
> #3.
> + are updated while the transactions are in progress. This means
> that we can see
> + those statistics being updated without having to wait until the transaction
> + finishes.
> + </para>
>
> The "This means ...... " line used several times does not add value, IMO.
> "are updated while the transactions are in progress." is sufficient.
Removed.
>
> #4.
> + <note>
> + <para>
> + All the statistics are updated while the transactions are in
> progress, except
> + for <structfield>xact_commit</structfield>,
> <structfield>xact_rollback</structfield>,
> + <structfield>tup_inserted</structfield>,
> <structfield>tup_updated</structfield> and
> + <structfield>tup_deleted</structfield> that are updated only when
> the transactions
> + finish.
> + </para>
> + </note>
>
> Only these 5 fields from pgstat_relation_flush_anytime_cb, so only the below are
> "All the statistics are updated while the transactions are in progress", right?
>
> numscans
> tuples_returned
> tuples_fetched
> blocks_fetched
> blocks_hit
No, 0003 also changes the flush mode for the database KIND. All the fields that
I mentioned are inherited from relations stats and are flushed only at transaction
boundaries (so they don't appear in pg_stat_database until the transaction
finishes). Does that make sense? (if the database kind is not switched to
flush any time then none would appear while the transaction is in progress, even
the ones inherited from relations stats).
PFA v3, also taking care of Zsolt's comment (thanks!) done up-thread.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v3-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (16.2K, ../../aXCroXBLiUpxk0Ap@ip-10-97-1-34.eu-west-3.compute.internal/2-v3-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From b9652e6e6031ff88f9b789f99c5c19a38fa61d0c Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v3 1/3] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second to call this function, ensuring timely stats visibility
without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling
pgstat_report_anytime_stat(false)
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/tcop/postgres.c | 16 ++++
src/backend/utils/activity/pgstat.c | 111 +++++++++++++++++++++++-----
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 15 ++++
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 4 +
src/include/utils/pgstat_internal.h | 16 ++++
src/include/utils/timeout.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 148 insertions(+), 18 deletions(-)
8.1% src/backend/tcop/
68.4% src/backend/utils/activity/
9.6% src/backend/utils/init/
9.2% src/include/utils/
4.1% src/include/
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e54bf1e760f..132fae61423 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3530,6 +3530,22 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+
+ /* Schedule next timeout */
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
+ PGSTAT_MIN_INTERVAL);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..ab4d9088a9a 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -122,8 +122,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +185,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +322,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +338,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +356,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +374,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -388,6 +393,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
.shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
@@ -404,6 +410,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter),
.shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
@@ -420,6 +427,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer),
.shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
@@ -436,6 +444,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +462,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +480,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +786,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1345,9 +1344,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,6 +1381,20 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
@@ -1402,6 +1420,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2164,33 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /*
+ * Exit if no pending stats at all. This avoids unnecessary work when
+ * backends are idle or in sessions without stats accumulation.
+ *
+ * Note: This check isn't precise as there might be only transactional
+ * stats pending, which we'll skip during the flush. However, maintaining
+ * precise tracking would add complexity that does not seem worth it from
+ * a performance point of view (no noticeable performance regression has
+ * been observed with the current implementation).
+ */
+ if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed)
+ return;
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..6076f531c4a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void);
static void IdleSessionTimeoutHandler(void);
static void IdleStatsUpdateTimeoutHandler(void);
static void ClientCheckTimeoutHandler(void);
+static void AnytimeStatsUpdateTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
}
/*
@@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void)
return result;
}
+
+/*
+ * Timeout handler for flushing non-transactional stats.
+ */
+static void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..8aeb9628871 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..1651f16f966 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,6 +536,7 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
extern void pgstat_reset_counters(void);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..46ce90c9624 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,9 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /* Flush mode */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3f3a888fd0e..d3912b43fdc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v3-0002-Remove-useless-calls-to-flush-some-stats.patch (6.4K, ../../aXCroXBLiUpxk0Ap@ip-10-97-1-34.eu-west-3.compute.internal/3-v3-0002-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From 6bce329c6597b7bbf72dfa1674e1f0e796a7621d Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v3 2/3] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
3 files changed, 2 insertions(+), 57 deletions(-)
75.3% src/backend/replication/
24.6% src/backend/utils/activity/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index a41453530a1..266379c780a 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -553,16 +553,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1ab09655a70..c33185bd337 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1826,7 +1822,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1847,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1958,8 +1952,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1968,15 +1961,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2879,8 +2863,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2975,9 +2957,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2985,7 +2964,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2996,21 +2974,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
--
2.34.1
[text/x-diff] v3-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch (17.0K, ../../aXCroXBLiUpxk0Ap@ip-10-97-1-34.eu-west-3.compute.internal/4-v3-0003-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch)
download | inline diff:
From 121f38f0fd1105a15d7acf8c25db23a4196b84da Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v3 3/3] Add FLUSH_MIXED support and implement it for RELATION
stats
This commit extends the non transactional stats infrastructure to support statistics
kinds with mixed transaction behavior: some fields are transactional (e.g., tuple
inserts/updates/deletes) while others are non transactional (e.g., sequential scans
blocks read, ...).
It introduces FLUSH_MIXED as a third flush mode type, alongside FLUSH_ANYTIME
and FLUSH_AT_TXN_BOUNDARY. For FLUSH_MIXED kinds, a new flush_anytime_cb callback
enables partial flushing of only the non transactional fields during running
transactions.
Some tests are also added.
Implementation details:
- Add FLUSH_MIXED to PgStat_FlushMode enum
- Add flush_anytime_cb to PgStat_KindInfo for partial flushing callback
- Update pgstat_flush_pending_entries() to call flush_anytime_cb for
FLUSH_MIXED entries when in anytime_only mode
- Keep FLUSH_MIXED entries in the pending list after partial flush, as
transactional fields still need to be flushed at transaction boundary
RELATION stats are making use of FLUSH_MIXED:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_MIXED
- Implement pgstat_relation_flush_anytime_cb() to flush only read related
stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
blocks_hit
- Clear these fields after flushing to prevent double counting when
pgstat_relation_flush_cb() runs at transaction commit
- Transactional stats (tuples_inserted, tuples_updated, tuples_deleted,
live_tuples, dead_tuples) remain pending until transaction boundary
The DATABASE kind is also changed from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME, so
that some stats inherited from relations stats are also visible while the transaction
is in progress.
Remark:
We could also imagine adding a new flush_anytime_static_cb() callback for
future FLUSH_MIXED fixed amount stats.
---
doc/src/sgml/monitoring.sgml | 26 +++++++
src/backend/utils/activity/pgstat.c | 32 ++++++--
src/backend/utils/activity/pgstat_relation.c | 78 ++++++++++++++++++++
src/include/utils/pgstat_internal.h | 9 +++
src/test/isolation/expected/stats.out | 40 ++++++++++
src/test/isolation/expected/stats_1.out | 40 ++++++++++
src/test/isolation/specs/stats.spec | 12 +++
7 files changed, 229 insertions(+), 8 deletions(-)
12.4% doc/src/sgml/
47.3% src/backend/utils/activity/
4.8% src/include/utils/
31.0% src/test/isolation/expected/
4.2% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 817fd9f4ca7..94fd2b76136 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3730,6 +3730,16 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ All the statistics are updated while the transactions are in progress, except
+ for <structfield>xact_commit</structfield>, <structfield>xact_rollback</structfield>,
+ <structfield>tup_inserted</structfield>, <structfield>tup_updated</structfield> and
+ <structfield>tup_deleted</structfield> that are updated only when the transactions
+ finish.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -4186,6 +4196,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4367,6 +4386,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ab4d9088a9a..6733a739c56 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -287,7 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,7 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_MIXED,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -313,6 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.pending_size = sizeof(PgStat_TableStatus),
.flush_pending_cb = pgstat_relation_flush_cb,
+ .flush_anytime_cb = pgstat_relation_flush_anytime_cb,
.delete_pending_cb = pgstat_relation_delete_pending_cb,
.reset_timestamp_cb = pgstat_relation_reset_timestamp_cb,
},
@@ -1345,10 +1346,11 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME and FLUSH_MIXED entries,
+ * using flush_anytime_cb for FLUSH_MIXED.
* This is safe to call inside transactions.
*
- * If anytime_only is false, flushes all entries.
+ * If anytime_only is false, flushes all entries using flush_pending_cb.
*/
static bool
pgstat_flush_pending_entries(bool nowait, bool anytime_only)
@@ -1376,6 +1378,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1395,8 +1398,21 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
continue;
}
- /* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ /* flush the stats (with the appropriate callback), if possible */
+ if (anytime_only &&
+ kind_info->flush_mode == FLUSH_MIXED &&
+ kind_info->flush_anytime_cb != NULL)
+ {
+ /* Partial flush of non-transactional fields only */
+ did_flush = kind_info->flush_anytime_cb(entry_ref, nowait);
+ is_partial_flush = true;
+ }
+ else
+ {
+ /* Full flush */
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ is_partial_flush = false;
+ }
Assert(did_flush || nowait);
@@ -1406,8 +1422,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index feae2ae5f44..d6b799c4354 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -887,6 +887,84 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+/*
+ * Flush only non-transactional relation stats.
+ *
+ * This is called periodically during running transactions to make some
+ * statistics visible without waiting for the transaction to finish.
+ *
+ * Transactional stats (inserts/updates/deletes and their effects on live/dead
+ * tuple counts) remain in pending until the transaction ends, at which point
+ * pgstat_relation_flush_cb() will flush them.
+ *
+ * If nowait is true and the lock could not be immediately acquired, returns
+ * false without flushing the entry. Otherwise returns true.
+ */
+bool
+pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ Oid dboid;
+ PgStat_TableStatus *lstats; /* pending stats entry */
+ PgStatShared_Relation *shtabstats;
+ PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
+ PgStat_StatDBEntry *dbentry; /* pending database entry */
+
+ dboid = entry_ref->shared_entry->key.dboid;
+ lstats = (PgStat_TableStatus *) entry_ref->pending;
+ shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (!(lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0))
+ return true;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ /* Add only the non-transactional values to the shared entry */
+ tabentry = &shtabstats->stats;
+
+ tabentry->numscans += lstats->counts.numscans;
+ if (lstats->counts.numscans)
+ {
+ TimestampTz t = GetCurrentTimestamp();
+
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+ tabentry->tuples_returned += lstats->counts.tuples_returned;
+ tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+ tabentry->blocks_fetched += lstats->counts.blocks_fetched;
+ tabentry->blocks_hit += lstats->counts.blocks_hit;
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ dbentry->tuples_returned += lstats->counts.tuples_returned;
+ dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ dbentry->blocks_fetched += lstats->counts.blocks_fetched;
+ dbentry->blocks_hit += lstats->counts.blocks_hit;
+
+ /*
+ * Clear the flushed fields from pending stats to prevent double-counting
+ * when pgstat_relation_flush_cb() runs at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
+}
+
void
pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref)
{
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 46ce90c9624..5f339f6d2ef 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -236,6 +236,8 @@ typedef enum PgStat_FlushMode
* transaction boundary */
FLUSH_ANYTIME, /* All fields can be flushed anytime,
* including within transactions */
+ FLUSH_MIXED, /* Mix of fields that can be flushed anytime
+ * or only at transaction boundary */
} PgStat_FlushMode;
/*
@@ -267,6 +269,12 @@ typedef struct PgStat_KindInfo
/* Flush mode */
PgStat_FlushMode flush_mode;
+ /*
+ * For FLUSH_MIXED kinds: callback to flush only some fields. If NULL for
+ * a MIXED kind, treated as FLUSH_AT_TXN_BOUNDARY.
+ */
+ bool (*flush_anytime_cb) (PgStat_EntryRef *entry_ref, bool nowait);
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -779,6 +787,7 @@ extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..6d62b30e4a7 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..2fade10e817 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 0| 0| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..1b0168e6176 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than PGSTAT_ANYTIME_FLUSH_INTERVAL
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -138,6 +140,7 @@ step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -435,6 +438,15 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-21 23:41 ` Sami Imseih <samimseih@gmail.com>
2026-01-22 09:52 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-01-21 23:41 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Thanks for the updated patches!
> No, 0003 also changes the flush mode for the database KIND. All the fields that
> I mentioned are inherited from relations stats and are flushed only at transaction
> boundaries (so they don't appear in pg_stat_database until the transaction
> finishes). Does that make sense?
yes, I understand it clearly now.
But, the Note under pg_stat_database reads like this:
"All the statistics are updated while the transactions are in progress,
except for xact_commit, xact_rollback, tup_inserted, tup_updated
and tup_deleted that are updated only when the transactions finish."
But that is not true for all pg_stat_database fields, such as session_time,
active_time, idle_in_transaction_time, etc. From what I can tell some of their
fields are updated when the connection is closed. For example
in one session run "select pg_sleep(10)" and in another session monitor
pg_stat_database.active_time. That will not be updated until the session
is closed.
This is because these are not relation stats, which makes sense. The
Note section should elaborate more on this, right?
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-21 23:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-22 09:52 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 17:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-22 09:52 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Wed, Jan 21, 2026 at 05:41:13PM -0600, Sami Imseih wrote:
> Thanks for the updated patches!
>
> > No, 0003 also changes the flush mode for the database KIND. All the fields that
> > I mentioned are inherited from relations stats and are flushed only at transaction
> > boundaries (so they don't appear in pg_stat_database until the transaction
> > finishes). Does that make sense?
>
> yes, I understand it clearly now.
>
> But, the Note under pg_stat_database reads like this:
>
> "All the statistics are updated while the transactions are in progress,
> except for xact_commit, xact_rollback, tup_inserted, tup_updated
> and tup_deleted that are updated only when the transactions finish."
>
> But that is not true for all pg_stat_database fields, such as session_time,
> active_time, idle_in_transaction_time, etc. From what I can tell some of their
> fields are updated when the connection is closed. For example
> in one session run "select pg_sleep(10)" and in another session monitor
> pg_stat_database.active_time. That will not be updated until the session
> is closed.
>
> This is because these are not relation stats, which makes sense. The
> Note section should elaborate more on this, right?
Yeah, so, while pgstat_database_flush_cb() is now called every second (if there
are pending stats), not all the stats would have their pending entries updated.
For example, pgstat_update_dbstats() updates some of them: xact_commit, xact_rollback,
blk_read_time, blk_write_time, session_time, active_time and idle_in_transaction_time
but only at transaction boundaries. Indeed, pgstat_update_dbstats() is only called
during pgstat_report_stat() and not during pgstat_report_anytime_stat().
I think that we could:
1. Update the doc as you suggest
or
2. Call a modified version of pgstat_update_dbstats() in pgstat_report_anytime_stat()
that would update blk_read_time, blk_write_time, session_time, active_time and
idle_in_transaction_time but that would require an extra GetCurrentTimestamp()
call.
or
3. Call a modified version of pgstat_update_dbstats() in pgstat_report_anytime_stat()
that would update the same as in 2. except session_time then avoiding the need
of a GetCurrentTimestamp() extra call.
I'm tempted to vote for 1. as I'm not sure of the added value of 2. and 3.,
thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-21 23:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 09:52 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-22 17:23 ` Sami Imseih <samimseih@gmail.com>
2026-01-26 07:35 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-01-22 17:23 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> For example, pgstat_update_dbstats() updates some of them: xact_commit, xact_rollback,
> blk_read_time, blk_write_time, session_time, active_time and idle_in_transaction_time
> but only at transaction boundaries. Indeed, pgstat_update_dbstats() is only called
> during pgstat_report_stat() and not during pgstat_report_anytime_stat().
>
> I think that we could:
>
> 1. Update the doc as you suggest
I am thinking the _time related fields are OK to be non-anytime
fields, since they
have overhead and also they can be actively monitored from pg_stat_activity
if someone really needs real time information.
The other session related counters don't need need special consideration.
parallel counters are anytime.
So, the documentation can mention the _time related fields that are flushed
only at their appropriate times.
Maybe something general like this:
"Some statistics are updated while a transaction is in progress.
Statistics that either do
not depend on transactions or require transactional consistency are
updated only
when the transaction ends. Statistics that require transactional consistency
include xact_commit, xact_rollback, tup_inserted, tup_updated, and tup_deleted."
What do you think?
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-21 23:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 09:52 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 17:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-26 07:35 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-27 16:53 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-26 07:35 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Thu, Jan 22, 2026 at 11:23:29AM -0600, Sami Imseih wrote:
> > For example, pgstat_update_dbstats() updates some of them: xact_commit, xact_rollback,
> > blk_read_time, blk_write_time, session_time, active_time and idle_in_transaction_time
> > but only at transaction boundaries. Indeed, pgstat_update_dbstats() is only called
> > during pgstat_report_stat() and not during pgstat_report_anytime_stat().
> >
> > I think that we could:
> >
> > 1. Update the doc as you suggest
>
> I am thinking the _time related fields are OK to be non-anytime
> fields, since they
> have overhead and also they can be actively monitored from pg_stat_activity
> if someone really needs real time information.
>
> The other session related counters don't need need special consideration.
>
> parallel counters are anytime.
>
> So, the documentation can mention the _time related fields that are flushed
> only at their appropriate times.
>
> Maybe something general like this:
>
> "Some statistics are updated while a transaction is in progress.
> Statistics that either do
> not depend on transactions or require transactional consistency are
> updated only
> when the transaction ends. Statistics that require transactional consistency
> include xact_commit, xact_rollback, tup_inserted, tup_updated, and tup_deleted."
>
> What do you think?
Thanks!
That sounds good, maybe just change the first sentence to something like:
"
Some statistics are updated while a transaction is in progress (this includes
blks_read, blks_hit, tup_returned and tup_fetched).
"
Thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-21 23:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 09:52 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 17:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-26 07:35 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-27 16:53 ` Sami Imseih <samimseih@gmail.com>
0 siblings, 0 replies; 71+ messages in thread
From: Sami Imseih @ 2026-01-27 16:53 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> Some statistics are updated while a transaction is in progress (this includes
> blks_read, blks_hit, tup_returned and tup_fetched).
I am ok with this. Just a slight change.
"""
Some statistics are updated while a transaction is in progress
(for example, blks_read, blks_hit, tup_returned, and tup_fetched).
"""
The difference is "for example" makes it clear this is not a comprehensive list.
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-22 00:02 ` Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Michael Paquier @ 2026-01-22 00:02 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Wed, Jan 21, 2026 at 10:34:09AM +0000, Bertrand Drouvot wrote:
> No, 0003 also changes the flush mode for the database KIND. All the fields that
> I mentioned are inherited from relations stats and are flushed only at transaction
> boundaries (so they don't appear in pg_stat_database until the transaction
> finishes). Does that make sense? (if the database kind is not switched to
> flush any time then none would appear while the transaction is in progress, even
> the ones inherited from relations stats).
>
> PFA v3, also taking care of Zsolt's comment (thanks!) done up-thread.
While reading through 0001, I got to question on which properties
and/or assumptions of a stats kind one has to rely on to decide to
what flush_mode should be set. To put is simpler, why don't we just
do a periodic pgstat_report_stat(false) call that would flush all the
stats for all stats kinds based on the new timeout registered,
expanding a bit the flush we currently do when idle in
ProcessInterrupts()? It seems that one point of contention should be
that we should be careful with entries in the shmem hash table that
have been created in a transactional way, but we may already flush
them while we are in a transaction state, no? Are there any fields in
a stats kind that we do may not want to flush? If yes, it sounds to
me that it would be better to document these in the structures to
explain the reason why a flush mode is chosen over the other.
I am also not convinced that we have to be that aggressive with these
extra flushes. The target is long-running analytical queries, that
could take minutes or even hours. Using the same value as
PGSTAT_IDLE_INTERVAL (10s), perhaps renaming the value while on it,
would be a more natural fit. A 1s vs 10s report interval does not
really matter for long analytical queries, where I'd imagine data
being picked up on at least a 30s interval, at the shortest. Of
course, one may want to get a more "live" representation of the data
with more aggressive flushes, but is that really helpful for
long-running queries to have more granularity, stressing more the
shmem state?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aXFpAisDKy6g_cKx@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
@ 2026-01-22 01:41 ` Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 02:28 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 71+ messages in thread
From: Sami Imseih @ 2026-01-22 01:41 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> > No, 0003 also changes the flush mode for the database KIND. All the fields that
> > I mentioned are inherited from relations stats and are flushed only at transaction
> > boundaries (so they don't appear in pg_stat_database until the transaction
> > finishes). Does that make sense? (if the database kind is not switched to
> > flush any time then none would appear while the transaction is in progress, even
> > the ones inherited from relations stats).
> >
> > PFA v3, also taking care of Zsolt's comment (thanks!) done up-thread.
>
> While reading through 0001, I got to question on which properties
> and/or assumptions of a stats kind one has to rely on to decide to
> what flush_mode should be set. To put is simpler, why don't we just
> do a periodic pgstat_report_stat(false) call that would flush all the
> stats for all stats kinds based on the new timeout registered,
> expanding a bit the flush we currently do when idle in
> ProcessInterrupts()?
There are some important cases in which we would want to
distinguish between a "transaction boundary" flush vs an
"anytime" flush.
For example, xact_commit/rollback. I would want those
fields to be in sync with tuples_inserted/updated/deleted
to allow for accurate calculations like number of inserts
per commit, etc.
Another one would be n_mod_since_analyze, That should
only be updated after commit (or not after rollback). Otherwise,
it may throw autovanalyze threshold calculations way off. Same
for n_dead_tup and autovacuum.
> I am also not convinced that we have to be that aggressive with these
> extra flushes. The target is long-running analytical queries, that
> could take minutes or even hours. Using the same value as
> PGSTAT_IDLE_INTERVAL (10s),
PGSTAT_IDLE_INTERVAL is flushing an idle backend every 10 seconds
IIUC. So this value only applies when outside of a transaction.
> A 1s vs 10s report interval does not really matter for long analytical queries.
Sure, Bertrand mentioned early in the thread that the anytime flushes
could be made configurable. Perhaps that is a good idea where we can
default with something large like 10s intervals for anytime flushes, but allow
the user to configure a more frequent flushes ( although I would think
that 1 sec is the minimum we should allow ).
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-22 01:56 ` Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Fujii Masao @ 2026-01-22 01:56 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Thu, Jan 22, 2026 at 10:41 AM Sami Imseih <samimseih@gmail.com> wrote:
>
> > > No, 0003 also changes the flush mode for the database KIND. All the fields that
> > > I mentioned are inherited from relations stats and are flushed only at transaction
> > > boundaries (so they don't appear in pg_stat_database until the transaction
> > > finishes). Does that make sense? (if the database kind is not switched to
> > > flush any time then none would appear while the transaction is in progress, even
> > > the ones inherited from relations stats).
> > >
> > > PFA v3, also taking care of Zsolt's comment (thanks!) done up-thread.
> >
> > While reading through 0001, I got to question on which properties
> > and/or assumptions of a stats kind one has to rely on to decide to
> > what flush_mode should be set. To put is simpler, why don't we just
> > do a periodic pgstat_report_stat(false) call that would flush all the
> > stats for all stats kinds based on the new timeout registered,
> > expanding a bit the flush we currently do when idle in
> > ProcessInterrupts()?
>
> There are some important cases in which we would want to
> distinguish between a "transaction boundary" flush vs an
> "anytime" flush.
>
> For example, xact_commit/rollback. I would want those
> fields to be in sync with tuples_inserted/updated/deleted
> to allow for accurate calculations like number of inserts
> per commit, etc.
>
> Another one would be n_mod_since_analyze, That should
> only be updated after commit (or not after rollback). Otherwise,
> it may throw autovanalyze threshold calculations way off. Same
> for n_dead_tup and autovacuum.
>
> > I am also not convinced that we have to be that aggressive with these
> > extra flushes. The target is long-running analytical queries, that
> > could take minutes or even hours. Using the same value as
> > PGSTAT_IDLE_INTERVAL (10s),
>
> PGSTAT_IDLE_INTERVAL is flushing an idle backend every 10 seconds
> IIUC. So this value only applies when outside of a transaction.
>
> > A 1s vs 10s report interval does not really matter for long analytical queries.
>
> Sure, Bertrand mentioned early in the thread that the anytime flushes
> could be made configurable. Perhaps that is a good idea where we can
> default with something large like 10s intervals for anytime flushes, but allow
> the user to configure a more frequent flushes ( although I would think
> that 1 sec is the minimum we should allow ).
+1 on adding an option to control the interval. With a fixed interval
(for example, 1s), log_lock_waits messages could be emitted that frequently,
which may be annoying for some users.
Of course, it would be even better if these periodic wakeups did not trigger
log_lock_waits messages at all, though.
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
@ 2026-01-22 07:43 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-22 07:43 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Thu, Jan 22, 2026 at 10:56:48AM +0900, Fujii Masao wrote:
> On Thu, Jan 22, 2026 at 10:41 AM Sami Imseih <samimseih@gmail.com> wrote:
> >
> > Sure, Bertrand mentioned early in the thread that the anytime flushes
> > could be made configurable. Perhaps that is a good idea where we can
> > default with something large like 10s intervals for anytime flushes, but allow
> > the user to configure a more frequent flushes ( although I would think
> > that 1 sec is the minimum we should allow ).
>
> +1 on adding an option to control the interval. With a fixed interval
> (for example, 1s), log_lock_waits messages could be emitted that frequently,
> which may be annoying for some users.
>
> Of course, it would be even better if these periodic wakeups did not trigger
> log_lock_waits messages at all, though.
pgstat_report_anytime_stat() is called with the force parameter set to false,
means that the flushes are done with nowait = true means that LWLockConditionalAcquire()
is used. In that case, do you still see cases where log_lock_waits messages could
be triggered due to the new flush?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-22 12:12 ` Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
0 siblings, 2 replies; 71+ messages in thread
From: Fujii Masao @ 2026-01-22 12:12 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Thu, Jan 22, 2026 at 4:43 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Thu, Jan 22, 2026 at 10:56:48AM +0900, Fujii Masao wrote:
> > On Thu, Jan 22, 2026 at 10:41 AM Sami Imseih <samimseih@gmail.com> wrote:
> > >
> > > Sure, Bertrand mentioned early in the thread that the anytime flushes
> > > could be made configurable. Perhaps that is a good idea where we can
> > > default with something large like 10s intervals for anytime flushes, but allow
> > > the user to configure a more frequent flushes ( although I would think
> > > that 1 sec is the minimum we should allow ).
> >
> > +1 on adding an option to control the interval. With a fixed interval
> > (for example, 1s), log_lock_waits messages could be emitted that frequently,
> > which may be annoying for some users.
> >
> > Of course, it would be even better if these periodic wakeups did not trigger
> > log_lock_waits messages at all, though.
>
> pgstat_report_anytime_stat() is called with the force parameter set to false,
> means that the flushes are done with nowait = true means that LWLockConditionalAcquire()
> is used. In that case, do you still see cases where log_lock_waits messages could
> be triggered due to the new flush?
I haven't read the patch in detail yet, but after applying patch 0001 and
causing a lock wait (for example, using the steps below), I observed that
log_lock_waits messages are emitted every second.
[session 1]
create table tbl as select id from generate_series(1, 10) id;
begin;
select * from tbl where id = 1 for update;
[session 2]
begin;
select * from tbl where id = 1 for update;
With this setup, the following messages were logged once per second:
LOG: process 72199 still waiting for ShareLock on transaction 771
after 63034.119 ms
DETAIL: Process holding the lock: 72190. Wait queue: 72199.
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
@ 2026-01-22 16:45 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-22 16:45 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Thu, Jan 22, 2026 at 09:12:18PM +0900, Fujii Masao wrote:
> On Thu, Jan 22, 2026 at 4:43 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > pgstat_report_anytime_stat() is called with the force parameter set to false,
> > means that the flushes are done with nowait = true means that LWLockConditionalAcquire()
> > is used. In that case, do you still see cases where log_lock_waits messages could
> > be triggered due to the new flush?
>
> I haven't read the patch in detail yet, but after applying patch 0001 and
> causing a lock wait (for example, using the steps below), I observed that
> log_lock_waits messages are emitted every second.
>
> [session 1]
> create table tbl as select id from generate_series(1, 10) id;
> begin;
> select * from tbl where id = 1 for update;
>
> [session 2]
> begin;
> select * from tbl where id = 1 for update;
>
> With this setup, the following messages were logged once per second:
>
> LOG: process 72199 still waiting for ShareLock on transaction 771
> after 63034.119 ms
> DETAIL: Process holding the lock: 72190. Wait queue: 72199.
>
Thanks!
I see, the WaitLatch() in ProcSleep() is "woken up" every 1s due to the
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,...) being set unconditionally
in ProcessInterrupts(). We need to be more restrictive as to when to enable the
timeout, I'll fix in the next version.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-26 06:59 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-26 06:59 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Thu, Jan 22, 2026 at 04:45:31PM +0000, Bertrand Drouvot wrote:
> On Thu, Jan 22, 2026 at 09:12:18PM +0900, Fujii Masao wrote:
> >
> > With this setup, the following messages were logged once per second:
> >
> > LOG: process 72199 still waiting for ShareLock on transaction 771
> > after 63034.119 ms
> > DETAIL: Process holding the lock: 72190. Wait queue: 72199.
> >
>
> Thanks!
>
> I see, the WaitLatch() in ProcSleep() is "woken up" every 1s due to the
> enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,...) being set unconditionally
> in ProcessInterrupts(). We need to be more restrictive as to when to enable the
> timeout, I'll fix in the next version.
The attached, to apply on top of 0001, fix the issue. However it handles only the
WaitLatch in ProcSleep() case and I start to have concern about the others WaitLatch()
that would/could be "woken up" every 1s.
Using disable_timeout() and enable_timeout_after() in WaitEventSetWait() does not
look like a great answer to this concern, so I wonder if we should use a larger
flush frequency instead (as proposed up-thread), thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 063826ae576..7376dd1f316 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -1322,6 +1322,7 @@ ProcSleep(LOCALLOCK *locallock)
bool allow_autovacuum_cancel = true;
bool logged_recovery_conflict = false;
ProcWaitStatus myWaitStatus;
+ bool anytime_timeout_was_active = false;
/* The caller must've armed the on-error cleanup mechanism */
Assert(GetAwaitedLock() == locallock);
@@ -1398,6 +1399,10 @@ ProcSleep(LOCALLOCK *locallock)
standbyWaitStart = GetCurrentTimestamp();
}
+ anytime_timeout_was_active = get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT);
+ if (anytime_timeout_was_active)
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+
/*
* If somebody wakes us between LWLockRelease and WaitLatch, the latch
* will not wait. But a set latch does not necessarily mean that the lock
@@ -1661,6 +1666,9 @@ ProcSleep(LOCALLOCK *locallock)
}
} while (myWaitStatus == PROC_WAIT_STATUS_WAITING);
+ if (anytime_timeout_was_active)
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
/*
* Disable the timers, if they are still running. As in LockErrorCleanup,
* we must preserve the LOCK_TIMEOUT indicator flag: if a lock timeout has
Attachments:
[text/plain] fix_ProcSleep.txt (1.3K, ../../aXcQ0Bdff5OsUDWY@ip-10-97-1-34.eu-west-3.compute.internal/2-fix_ProcSleep.txt)
download | inline diff:
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 063826ae576..7376dd1f316 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -1322,6 +1322,7 @@ ProcSleep(LOCALLOCK *locallock)
bool allow_autovacuum_cancel = true;
bool logged_recovery_conflict = false;
ProcWaitStatus myWaitStatus;
+ bool anytime_timeout_was_active = false;
/* The caller must've armed the on-error cleanup mechanism */
Assert(GetAwaitedLock() == locallock);
@@ -1398,6 +1399,10 @@ ProcSleep(LOCALLOCK *locallock)
standbyWaitStart = GetCurrentTimestamp();
}
+ anytime_timeout_was_active = get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT);
+ if (anytime_timeout_was_active)
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+
/*
* If somebody wakes us between LWLockRelease and WaitLatch, the latch
* will not wait. But a set latch does not necessarily mean that the lock
@@ -1661,6 +1666,9 @@ ProcSleep(LOCALLOCK *locallock)
}
} while (myWaitStatus == PROC_WAIT_STATUS_WAITING);
+ if (anytime_timeout_was_active)
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
/*
* Disable the timers, if they are still running. As in LockErrorCleanup,
* we must preserve the LOCK_TIMEOUT indicator flag: if a lock timeout has
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-28 06:35 ` Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Michael Paquier @ 2026-01-28 06:35 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Sami Imseih <samimseih@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Mon, Jan 26, 2026 at 06:59:28AM +0000, Bertrand Drouvot wrote:
> The attached, to apply on top of 0001, fix the issue. However it handles only the
> WaitLatch in ProcSleep() case and I start to have concern about the others WaitLatch()
> that would/could be "woken up" every 1s.
Hmm, I indeed suspect that is may not be the only one.. This is much bigger.
> Using disable_timeout() and enable_timeout_after() in WaitEventSetWait() does not
> look like a great answer to this concern, so I wonder if we should use a larger
> flush frequency instead (as proposed up-thread), thoughts?
Only a larger frequency is not the correct answer here. It would just
reduce the frequency of the extra lock wait messages for one: these
should never appear more than necessary. And how about for example
extension code?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aXmuKZz8sXPLCumT@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
@ 2026-01-28 11:14 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-28 11:14 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Sami Imseih <samimseih@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Wed, Jan 28, 2026 at 03:35:21PM +0900, Michael Paquier wrote:
> On Mon, Jan 26, 2026 at 06:59:28AM +0000, Bertrand Drouvot wrote:
> > The attached, to apply on top of 0001, fix the issue. However it handles only the
> > WaitLatch in ProcSleep() case and I start to have concern about the others WaitLatch()
> > that would/could be "woken up" every 1s.
>
> > Using disable_timeout() and enable_timeout_after() in WaitEventSetWait() does not
> > look like a great answer to this concern, so I wonder if we should use a larger
> > flush frequency instead (as proposed up-thread), thoughts?
>
> Only a larger frequency is not the correct answer here. It would just
> reduce the frequency of the extra lock wait messages for one: these
> should never appear more than necessary.
Right. The fix in fix_ProcSleep.txt shared up-thread solves that and has been
added in 0001 attached.
Also the attached is now split in 4 sub-patches with 0002 introducing a new
GUC to control the flush interval (default is 10s). Note that 0001 to 0003 could
be merged as one patch but I did it that way to ease the review.
The new version also adds more documentation and takes care of Sami's comments
shared up-thread.
> And how about for example extension code?
I think that depending on how they write their code around WaitLatch (if any) they
could see messages being reported (if they do) in the logs at stats_flush_interval
frequency.
That said the default value is 10s and that looks pretty long for a latch to wait on.
In any case, they should already take care of a latch being woken by WL_LATCH_SET.
What do you think?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v4-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (17.9K, ../../aXnveIxQEB5wwcB8@ip-10-97-1-34.eu-west-3.compute.internal/2-v4-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From 07ac566644abd0ee01ab76cb96b0b176f3d25c5a Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v4 1/4] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second to call this function, ensuring timely stats visibility
without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling
pgstat_report_anytime_stat(false)
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/storage/lmgr/proc.c | 10 +++
src/backend/tcop/postgres.c | 16 ++++
src/backend/utils/activity/pgstat.c | 111 +++++++++++++++++++++++-----
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 15 ++++
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 4 +
src/include/utils/pgstat_internal.h | 20 +++++
src/include/utils/timeout.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 162 insertions(+), 18 deletions(-)
7.0% src/backend/storage/lmgr/
7.3% src/backend/tcop/
61.1% src/backend/utils/activity/
8.6% src/backend/utils/init/
11.8% src/include/utils/
3.6% src/include/
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 063826ae576..012705a2ee6 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -1322,6 +1322,7 @@ ProcSleep(LOCALLOCK *locallock)
bool allow_autovacuum_cancel = true;
bool logged_recovery_conflict = false;
ProcWaitStatus myWaitStatus;
+ bool anytime_timeout_was_active = false;
/* The caller must've armed the on-error cleanup mechanism */
Assert(GetAwaitedLock() == locallock);
@@ -1398,6 +1399,12 @@ ProcSleep(LOCALLOCK *locallock)
standbyWaitStart = GetCurrentTimestamp();
}
+ anytime_timeout_was_active = get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT);
+
+ /* No need to try to flush the statistics while the process is sleeping */
+ if (anytime_timeout_was_active)
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+
/*
* If somebody wakes us between LWLockRelease and WaitLatch, the latch
* will not wait. But a set latch does not necessarily mean that the lock
@@ -1661,6 +1668,9 @@ ProcSleep(LOCALLOCK *locallock)
}
} while (myWaitStatus == PROC_WAIT_STATUS_WAITING);
+ if (anytime_timeout_was_active)
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
/*
* Disable the timers, if they are still running. As in LockErrorCleanup,
* we must preserve the LOCK_TIMEOUT indicator flag: if a lock timeout has
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e54bf1e760f..132fae61423 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3530,6 +3530,22 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+
+ /* Schedule next timeout */
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
+ PGSTAT_MIN_INTERVAL);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..ab4d9088a9a 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -122,8 +122,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +185,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +322,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +338,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +356,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +374,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -388,6 +393,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
.shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
@@ -404,6 +410,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter),
.shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
@@ -420,6 +427,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer),
.shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
@@ -436,6 +444,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +462,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +480,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +786,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1345,9 +1344,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,6 +1381,20 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
@@ -1402,6 +1420,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2164,33 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /*
+ * Exit if no pending stats at all. This avoids unnecessary work when
+ * backends are idle or in sessions without stats accumulation.
+ *
+ * Note: This check isn't precise as there might be only transactional
+ * stats pending, which we'll skip during the flush. However, maintaining
+ * precise tracking would add complexity that does not seem worth it from
+ * a performance point of view (no noticeable performance regression has
+ * been observed with the current implementation).
+ */
+ if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed)
+ return;
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..6076f531c4a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void);
static void IdleSessionTimeoutHandler(void);
static void IdleStatsUpdateTimeoutHandler(void);
static void ClientCheckTimeoutHandler(void);
+static void AnytimeStatsUpdateTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
}
/*
@@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void)
return result;
}
+
+/*
+ * Timeout handler for flushing non-transactional stats.
+ */
+static void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..8aeb9628871 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..1651f16f966 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,6 +536,7 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
extern void pgstat_reset_counters(void);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..9ca39ea9a9a 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,13 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * Some stats have to be updated only at transaction boundaries (such as
+ * tuples_inserted updated, deleted), so it's very important to set the
+ * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default).
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ddbe4c64971..af21c87234a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v4-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch (8.3K, ../../aXnveIxQEB5wwcB8@ip-10-97-1-34.eu-west-3.compute.internal/3-v4-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch)
download | inline diff:
From 02ce5e90269558cf481a49e557509eed68d6a9dc Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v4 2/4] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 32 +++++++++++++++++++
src/backend/storage/lmgr/proc.c | 2 +-
src/backend/tcop/postgres.c | 2 +-
src/backend/utils/activity/pgstat.c | 15 +++++++++
src/backend/utils/init/postinit.c | 2 +-
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/pgstat.h | 1 +
src/include/utils/guc_hooks.h | 1 +
9 files changed, 63 insertions(+), 3 deletions(-)
55.5% doc/src/sgml/
5.3% src/backend/storage/lmgr/
12.6% src/backend/utils/activity/
5.3% src/backend/utils/init/
14.9% src/backend/utils/misc/
3.9% src/include/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5560b95ee60..3136816a933 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8834,6 +8834,38 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which non-transactional statistics are made visible
+ during running transactions. Non-transactional statistics include, for
+ example, WAL activity and I/O operations.
+ They become visible at that interval in monitoring views such as
+ <link linkend="monitoring-pg-stat-io-view"> <structname>pg_stat_io</structname></link>
+ and <link linkend="monitoring-pg-stat-wal-view"> <structname>pg_stat_wal</structname></link>
+ during running transactions.
+ If this value is specified without units, it is taken as milliseconds.
+ The default is 10 seconds (<literal>10s</literal>), which is probably
+ about the smallest value you would want in practice for long running
+ transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect transactional statistics such as
+ <structname>pg_stat_all_tables</structname> columns (like
+ <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ <structfield>n_tup_del</structfield>), which are always flushed at transaction
+ boundaries to maintain consistency.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 012705a2ee6..caa6eecca88 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -1669,7 +1669,7 @@ ProcSleep(LOCALLOCK *locallock)
} while (myWaitStatus == PROC_WAIT_STATUS_WAITING);
if (anytime_timeout_was_active)
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
/*
* Disable the timers, if they are still running. As in LockErrorCleanup,
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 132fae61423..c0e81cb13d0 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3543,7 +3543,7 @@ ProcessInterrupts(void)
/* Schedule next timeout */
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
- PGSTAT_MIN_INTERVAL);
+ pgstat_flush_interval);
}
if (ProcSignalBarrierPending)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ab4d9088a9a..ca08dd49cd7 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -113,6 +113,7 @@
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
/* ----------
@@ -202,6 +203,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -2165,6 +2167,19 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ {
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+ }
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 6076f531c4a..c7c0d618671 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -768,7 +768,7 @@ InitPostgres(const char *in_dbname, Oid dboid,
IdleStatsUpdateTimeoutHandler);
RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
AnytimeStatsUpdateTimeoutHandler);
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
}
/*
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index f0260e6e412..3bb43362e51 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2782,6 +2782,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c4f92fcdac8..6ce5a250170 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -669,6 +669,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 1651f16f966..e0f222695bf 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -816,6 +816,7 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index b6ecb0e769f..3a2ae6c41cd 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
--
2.34.1
[text/x-diff] v4-0003-Remove-useless-calls-to-flush-some-stats.patch (7.6K, ../../aXnveIxQEB5wwcB8@ip-10-97-1-34.eu-west-3.compute.internal/4-v4-0003-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From fe85fd375c89a85a84e47a88f01a9d629a8abb09 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v4 3/4] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.9% src/backend/replication/
22.8% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 6970af3f3ff..dcbe3517b46 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -565,16 +565,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a0e6a3d200c..74102def9c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1825,7 +1821,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1846,7 +1841,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1957,8 +1951,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1967,15 +1960,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2878,8 +2862,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2974,9 +2956,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2984,7 +2963,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2995,21 +2973,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..c058a5f9b1f 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index d7e62e4d488..dda872f7074 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v4-0004-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch (17.5K, ../../aXnveIxQEB5wwcB8@ip-10-97-1-34.eu-west-3.compute.internal/5-v4-0004-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch)
download | inline diff:
From fe3d6b69d8a9eb91d74c195c0d8617096ef56297 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v4 4/4] Add FLUSH_MIXED support and implement it for RELATION
stats
This commit extends the non transactional stats infrastructure to support statistics
kinds with mixed transaction behavior: some fields are transactional (e.g., tuple
inserts/updates/deletes) while others are non transactional (e.g., sequential scans
blocks read, ...).
It introduces FLUSH_MIXED as a third flush mode type, alongside FLUSH_ANYTIME
and FLUSH_AT_TXN_BOUNDARY. For FLUSH_MIXED kinds, a new flush_anytime_cb callback
enables partial flushing of only the non transactional fields during running
transactions.
Some tests are also added.
Implementation details:
- Add FLUSH_MIXED to PgStat_FlushMode enum
- Add flush_anytime_cb to PgStat_KindInfo for partial flushing callback
- Update pgstat_flush_pending_entries() to call flush_anytime_cb for
FLUSH_MIXED entries when in anytime_only mode
- Keep FLUSH_MIXED entries in the pending list after partial flush, as
transactional fields still need to be flushed at transaction boundary
RELATION stats are making use of FLUSH_MIXED:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_MIXED
- Implement pgstat_relation_flush_anytime_cb() to flush only read related
stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
blocks_hit
- Clear these fields after flushing to prevent double counting when
pgstat_relation_flush_cb() runs at transaction commit
- Transactional stats (tuples_inserted, tuples_updated, tuples_deleted,
live_tuples, dead_tuples) remain pending until transaction boundary
The DATABASE kind is also changed from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME, so
that some stats inherited from relations stats are also visible while the transaction
is in progress.
Remark:
We could also imagine adding a new flush_anytime_static_cb() callback for
future FLUSH_MIXED fixed amount stats.
---
doc/src/sgml/monitoring.sgml | 29 ++++++++
src/backend/utils/activity/pgstat.c | 32 ++++++--
src/backend/utils/activity/pgstat_relation.c | 78 ++++++++++++++++++++
src/include/utils/pgstat_internal.h | 9 +++
src/test/isolation/expected/stats.out | 40 ++++++++++
src/test/isolation/expected/stats_1.out | 40 ++++++++++
src/test/isolation/specs/stats.spec | 17 ++++-
7 files changed, 236 insertions(+), 9 deletions(-)
15.3% doc/src/sgml/
44.8% src/backend/utils/activity/
4.6% src/include/utils/
29.4% src/test/isolation/expected/
5.6% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..581d6ea7811 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ca08dd49cd7..987bf1c0f6e 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -289,7 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -307,7 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_MIXED,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -315,6 +315,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.pending_size = sizeof(PgStat_TableStatus),
.flush_pending_cb = pgstat_relation_flush_cb,
+ .flush_anytime_cb = pgstat_relation_flush_anytime_cb,
.delete_pending_cb = pgstat_relation_delete_pending_cb,
.reset_timestamp_cb = pgstat_relation_reset_timestamp_cb,
},
@@ -1347,10 +1348,11 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME and FLUSH_MIXED entries,
+ * using flush_anytime_cb for FLUSH_MIXED.
* This is safe to call inside transactions.
*
- * If anytime_only is false, flushes all entries.
+ * If anytime_only is false, flushes all entries using flush_pending_cb.
*/
static bool
pgstat_flush_pending_entries(bool nowait, bool anytime_only)
@@ -1378,6 +1380,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1397,8 +1400,21 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
continue;
}
- /* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ /* flush the stats (with the appropriate callback), if possible */
+ if (anytime_only &&
+ kind_info->flush_mode == FLUSH_MIXED &&
+ kind_info->flush_anytime_cb != NULL)
+ {
+ /* Partial flush of non-transactional fields only */
+ did_flush = kind_info->flush_anytime_cb(entry_ref, nowait);
+ is_partial_flush = true;
+ }
+ else
+ {
+ /* Full flush */
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ is_partial_flush = false;
+ }
Assert(did_flush || nowait);
@@ -1408,8 +1424,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index feae2ae5f44..d6b799c4354 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -887,6 +887,84 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+/*
+ * Flush only non-transactional relation stats.
+ *
+ * This is called periodically during running transactions to make some
+ * statistics visible without waiting for the transaction to finish.
+ *
+ * Transactional stats (inserts/updates/deletes and their effects on live/dead
+ * tuple counts) remain in pending until the transaction ends, at which point
+ * pgstat_relation_flush_cb() will flush them.
+ *
+ * If nowait is true and the lock could not be immediately acquired, returns
+ * false without flushing the entry. Otherwise returns true.
+ */
+bool
+pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ Oid dboid;
+ PgStat_TableStatus *lstats; /* pending stats entry */
+ PgStatShared_Relation *shtabstats;
+ PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
+ PgStat_StatDBEntry *dbentry; /* pending database entry */
+
+ dboid = entry_ref->shared_entry->key.dboid;
+ lstats = (PgStat_TableStatus *) entry_ref->pending;
+ shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (!(lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0))
+ return true;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ /* Add only the non-transactional values to the shared entry */
+ tabentry = &shtabstats->stats;
+
+ tabentry->numscans += lstats->counts.numscans;
+ if (lstats->counts.numscans)
+ {
+ TimestampTz t = GetCurrentTimestamp();
+
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+ tabentry->tuples_returned += lstats->counts.tuples_returned;
+ tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+ tabentry->blocks_fetched += lstats->counts.blocks_fetched;
+ tabentry->blocks_hit += lstats->counts.blocks_hit;
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ dbentry->tuples_returned += lstats->counts.tuples_returned;
+ dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ dbentry->blocks_fetched += lstats->counts.blocks_fetched;
+ dbentry->blocks_hit += lstats->counts.blocks_hit;
+
+ /*
+ * Clear the flushed fields from pending stats to prevent double-counting
+ * when pgstat_relation_flush_cb() runs at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
+}
+
void
pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref)
{
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9ca39ea9a9a..f91bbfd460f 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -236,6 +236,8 @@ typedef enum PgStat_FlushMode
* transaction boundary */
FLUSH_ANYTIME, /* All fields can be flushed anytime,
* including within transactions */
+ FLUSH_MIXED, /* Mix of fields that can be flushed anytime
+ * or only at transaction boundary */
} PgStat_FlushMode;
/*
@@ -271,6 +273,12 @@ typedef struct PgStat_KindInfo
*/
PgStat_FlushMode flush_mode;
+ /*
+ * For FLUSH_MIXED kinds: callback to flush only some fields. If NULL for
+ * a MIXED kind, treated as FLUSH_AT_TXN_BOUNDARY.
+ */
+ bool (*flush_anytime_cb) (PgStat_EntryRef *entry_ref, bool nowait);
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -783,6 +791,7 @@ extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..6d62b30e4a7 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..2fade10e817 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 0| 0| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..a4084efda49 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -435,6 +441,15 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-31 01:33 ` Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-01-31 01:33 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> Also the attached is now split in 4 sub-patches with 0002 introducing a new
> GUC to control the flush interval (default is 10s). Note that 0001 to 0003 could
> be merged as one patch but I did it that way to ease the review.
Thanks for the patches!
I do like the GUC introduced in 0002 to control the ANYTIME stats
flush interval.
But, as I was looking at this a bit more today and testing it, I am
not quite sure
I like what is happening here:
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /*
+ * Exit if no pending stats at all. This avoids unnecessary work when
+ * backends are idle or in sessions without stats accumulation.
+ *
+ * Note: This check isn't precise as there might be only transactional
+ * stats pending, which we'll skip during the flush. However,
maintaining
+ * precise tracking would add complexity that does not seem
worth it from
+ * a performance point of view (no noticeable performance regression has
+ * been observed with the current implementation).
+ */
+ if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed)
+ return;
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
There is a check if pgStatPending is empty so we can return early.
However, with FLUSH_MIXED, the scenario is we will more than likely always
have TXN_BOUNDARY stats that can only be flushed at the end of a transaction.
This means that most of the time we will call pgstat_flush_pending_entries
and then leave it up to the anytime_cb to check for pending ANYTIME stats to
flush (before taking the lock ).
+bool
+pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ Oid dboid;
+ PgStat_TableStatus *lstats; /* pending stats entry */
+ PgStatShared_Relation *shtabstats;
+ PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
+ PgStat_StatDBEntry *dbentry; /* pending database entry */
+
+ dboid = entry_ref->shared_entry->key.dboid;
+ lstats = (PgStat_TableStatus *) entry_ref->pending;
+ shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (!(lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0))
+ return true;
This makes things confusing because instead of just relying on
dlist_is_empty(&pgStatPending) to check if we need to flush anything,
the responsibility now moves to the callback, which now also has to
account for all the ANYTIME fields.
The way I can think about making this better is to somehow track if
we have ANYTIME data that needs to be flushed a different way
( maybe a second pgStatPending list for anytime stats ?? )
What do you think?
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-31 17:16 ` Sami Imseih <samimseih@gmail.com>
2026-02-02 07:31 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 2 replies; 71+ messages in thread
From: Sami Imseih @ 2026-01-31 17:16 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> This makes things confusing because instead of just relying on
> dlist_is_empty(&pgStatPending) to check if we need to flush anything,
> the responsibility now moves to the callback, which now also has to
> account for all the ANYTIME fields.
>
> The way I can think about making this better is to somehow track if
> we have ANYTIME data that needs to be flushed a different way
> ( maybe a second pgStatPending list for anytime stats ?? )
Thinking about this a bit more, it seems the FLUSH_MIXED may
not be needed. Instead can we just introduce a new global variable
called pgstat_report_variable_anytime which acts like
pgstat_report_fixed, except it's set to true whenever we update
anytime variable-numbered stats
```
#define pgstat_count_heap_scan(rel)
\
do {
\
pgstat_report_variable_anytime = true;
\
if (pgstat_should_count_relation(rel))
\
(rel)->pgstat_info->counts.numscans++;
\
} while (0)
````
and reset whenever we call pgstat_report_anytime_stat, like this:
```
void
pgstat_report_anytime_stat(bool force)
{
.....
.........
if (!pgstat_report_variable_anytime && !pgstat_report_fixed)
return;
/* Flush stats outside of transaction boundary */
if (pgstat_report_variable_anytime)
pgstat_flush_pending_entries(nowait, true);
....
........
pgstat_report_variable_anytime = false;
}
```
and then inside pgstat_flush_pending_entries, we just look
for kind_info->flush_mode == FLUSH_ANYTIME to flush.
```
/* flush the stats (with the appropriate callback), if possible */
if (anytime_only &&
kind_info->flush_mode == FLUSH_ANYTIME &&
kind_info->flush_anytime_cb != NULL)
{
/* Partial flush of non-transactional fields only */
did_flush = kind_info->flush_anytime_cb(entry_ref, nowait);
is_partial_flush = true;
}
````
With this approach, we will not enter pgstat_flush_pending_entries
inside pgstat_report_anytime_stat unless we have anytime
variable stats to report.
Also, the anytime flush callback does not need to check if there are
any variable-numbered stats to flush. This will not be needed as
it is in v4-0004
```
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (!(lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0))
+ return true;
```
This feels like an easier approach to reason about and we don't
need to add a third flush mode.
Thoughts?
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-02 07:31 ` Michael Paquier <michael@paquier.xyz>
2026-02-02 17:19 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Michael Paquier @ 2026-02-02 07:31 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Sat, Jan 31, 2026 at 11:16:03AM -0600, Sami Imseih wrote:
> Also, the anytime flush callback does not need to check if there are
> any variable-numbered stats to flush. This will not be needed as
> it is in v4-0004
>
> ```
> + /*
> + * Check if there are any non-transactional stats to flush. Avoid
> + * unnecessarily locking the entry if nothing accumulated.
> + */
> + if (!(lstats->counts.numscans > 0 ||
> + lstats->counts.tuples_returned > 0 ||
> + lstats->counts.tuples_fetched > 0 ||
> + lstats->counts.blocks_fetched > 0 ||
> + lstats->counts.blocks_hit > 0))
> + return true;
> ```
I think that this kind of check may finish by being expensive.
Reducing the number of fields to check for a non-flushing state, if
required, would be preferrable.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aYBS5BghHtloMN6b@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 07:31 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
@ 2026-02-02 17:19 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-02 17:19 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Mon, Feb 02, 2026 at 04:31:48PM +0900, Michael Paquier wrote:
> On Sat, Jan 31, 2026 at 11:16:03AM -0600, Sami Imseih wrote:
> > Also, the anytime flush callback does not need to check if there are
> > any variable-numbered stats to flush. This will not be needed as
> > it is in v4-0004
> >
> > ```
> > + /*
> > + * Check if there are any non-transactional stats to flush. Avoid
> > + * unnecessarily locking the entry if nothing accumulated.
> > + */
> > + if (!(lstats->counts.numscans > 0 ||
> > + lstats->counts.tuples_returned > 0 ||
> > + lstats->counts.tuples_fetched > 0 ||
> > + lstats->counts.blocks_fetched > 0 ||
> > + lstats->counts.blocks_hit > 0))
> > + return true;
> > ```
>
> I think that this kind of check may finish by being expensive.
> Reducing the number of fields to check for a non-flushing state, if
> required, would be preferrable.
Done, in v5 just shared up-thread.
Oh and I forgot to mention that with the new design in v5, then the fix for
the issue discovered by Masao-san is not needed anymore (and so has been removed
from 0001).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-02 17:16 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 15:57 ` Re: Flush some statistics within running transactions Zsolt Parragi <zsolt.parragi@percona.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
1 sibling, 2 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-02 17:16 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Sat, Jan 31, 2026 at 11:16:03AM -0600, Sami Imseih wrote:
> > This makes things confusing because instead of just relying on
> > dlist_is_empty(&pgStatPending) to check if we need to flush anything,
> > the responsibility now moves to the callback, which now also has to
> > account for all the ANYTIME fields.
> >
>
> Thinking about this a bit more, it seems the FLUSH_MIXED may
> not be needed. Instead can we just introduce a new global variable
> called pgstat_report_variable_anytime which acts like
> pgstat_report_fixed, except it's set to true whenever we update
> anytime variable-numbered stats
>
> With this approach, we will not enter pgstat_flush_pending_entries
> inside pgstat_report_anytime_stat unless we have anytime
> variable stats to report.
Thanks for looking at it!
In v5 attached I changed the design so that we don't re-enable the timeout
after each stats flush in ProcessInterrupts(). Instead the timeout is enabled
when we set pgstat_report_fixed to true or in pgstat_prep_pending_entry() when
appropriate. So that we know that when we enter pgstat_report_anytime_stat() that's
for good reasons and we don't need extra checks in it.
> ```
> + /*
> + * Check if there are any non-transactional stats to flush. Avoid
> + * unnecessarily locking the entry if nothing accumulated.
> + */
> + if (!(lstats->counts.numscans > 0 ||
> + lstats->counts.tuples_returned > 0 ||
> + lstats->counts.tuples_fetched > 0 ||
> + lstats->counts.blocks_fetched > 0 ||
> + lstats->counts.blocks_hit > 0))
> + return true;
> ```
Yeah, with the new design in place then those are not needed anymore.
> This feels like an easier approach to reason about and we don't
> need to add a third flush mode.
I do think we still need it. Indeed in 0004 that helps distinguish between
anytime flush or mixed flush (with the help of the new pgstat_report_mixed_anytime
global variable) in pgstat_prep_pending_entry().
Thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v5-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (28.5K, ../../aYDb7g5hXnaP+5BU@ip-10-97-1-34.eu-west-3.compute.internal/2-v5-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From bddc98bb813e2a0453112b47431d324657eabaab Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v5 1/4] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling
pgstat_report_anytime_stat(false)
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 9 ++
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 118 ++++++++++++++++----
src/backend/utils/activity/pgstat_backend.c | 9 ++
src/backend/utils/activity/pgstat_io.c | 5 +
src/backend/utils/activity/pgstat_slru.c | 5 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 4 +
src/include/utils/pgstat_internal.h | 21 ++++
src/include/utils/timeout.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
19 files changed, 213 insertions(+), 25 deletions(-)
5.5% src/backend/access/transam/
15.3% src/backend/postmaster/
3.4% src/backend/replication/
3.8% src/backend/tcop/
57.0% src/backend/utils/activity/
9.2% src/include/utils/
5.3% src/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..9503aea5b4d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,10 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2070,11 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
+ PGSTAT_MIN_INTERVAL);
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 80e3088fc7e..ab5d0645026 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -104,7 +106,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -114,6 +116,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 6482c21b8f9..6e187315613 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -216,7 +217,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -226,6 +227,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index a1a4f65f9a9..498d147f0da 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -246,6 +247,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c3d56c866d3..cec5dfdb430 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -250,7 +252,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -272,6 +274,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 38ec8a4c8c7..7416ca703c9 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -107,7 +109,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -117,6 +119,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 8b99160ed0e..24d7ef795cb 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -253,7 +255,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -261,6 +263,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index b4a8d2f3a1c..d19aa45400d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3530,6 +3530,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..2c9454677e9 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -112,6 +112,7 @@
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +123,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -388,6 +394,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
.shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
@@ -404,6 +411,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter),
.shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
@@ -420,6 +428,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer),
.shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
@@ -436,6 +445,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +463,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +481,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +787,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,12 +1293,18 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
entry_ref->pending = MemoryContextAllocZero(pgStatPendingContext, entrysize);
dlist_push_tail(&pgStatPending, &entry_ref->pending_node);
+
+ /* Schedule next anytime stats update timeout */
+ if (kind_info->flush_mode == FLUSH_ANYTIME && IsUnderPostmaster &&
+ !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
}
return entry_ref;
@@ -1345,9 +1351,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,6 +1388,20 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
@@ -1402,6 +1427,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2171,31 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
+
+/*
+ * Timeout handler for flushing non-transactional stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 1350f5f62f1..9dcb24db975 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,10 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +87,10 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..53dbf2a514b 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,10 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
have_iostats = true;
pgstat_report_fixed = true;
}
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..1d16cde1889 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -223,6 +224,10 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..f45365f47f7 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -765,6 +766,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..8aeb9628871 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..1651f16f966 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,6 +536,7 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
extern void pgstat_reset_counters(void);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..a9190078d0e 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,13 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * Some stats have to be updated only at transaction boundaries (such as
+ * tuples_inserted updated, deleted), so it's very important to set the
+ * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default).
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -677,6 +697,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9f5ee8fd482..860f835c088 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v5-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch (10.5K, ../../aYDb7g5hXnaP+5BU@ip-10-97-1-34.eu-west-3.compute.internal/3-v5-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch)
download | inline diff:
From c64a3a582201fc7d1925d374ad6fa5778b0d4d14 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v5 2/4] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 32 +++++++++++++++++++
src/backend/access/transam/xlog.c | 4 +--
src/backend/utils/activity/pgstat.c | 16 +++++++++-
src/backend/utils/activity/pgstat_backend.c | 4 +--
src/backend/utils/activity/pgstat_io.c | 2 +-
src/backend/utils/activity/pgstat_slru.c | 2 +-
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/pgstat.h | 1 +
src/include/utils/guc_hooks.h | 1 +
10 files changed, 66 insertions(+), 7 deletions(-)
45.9% doc/src/sgml/
6.5% src/backend/access/transam/
31.8% src/backend/utils/activity/
12.3% src/backend/utils/misc/
3.2% src/include/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5560b95ee60..3136816a933 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8834,6 +8834,38 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which non-transactional statistics are made visible
+ during running transactions. Non-transactional statistics include, for
+ example, WAL activity and I/O operations.
+ They become visible at that interval in monitoring views such as
+ <link linkend="monitoring-pg-stat-io-view"> <structname>pg_stat_io</structname></link>
+ and <link linkend="monitoring-pg-stat-wal-view"> <structname>pg_stat_wal</structname></link>
+ during running transactions.
+ If this value is specified without units, it is taken as milliseconds.
+ The default is 10 seconds (<literal>10s</literal>), which is probably
+ about the smallest value you would want in practice for long running
+ transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect transactional statistics such as
+ <structname>pg_stat_all_tables</structname> columns (like
+ <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ <structfield>n_tup_del</structfield>), which are always flushed at transaction
+ boundaries to maintain consistency.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 9503aea5b4d..31523dea923 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1087,7 +1087,7 @@ XLogInsertRecord(XLogRecData *rdata,
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
@@ -2073,7 +2073,7 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
- PGSTAT_MIN_INTERVAL);
+ pgstat_flush_interval);
/*
* Required for the flush of pending stats WAL data, per
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 2c9454677e9..dd174129403 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -203,6 +203,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -1304,7 +1305,7 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
/* Schedule next anytime stats update timeout */
if (kind_info->flush_mode == FLUSH_ANYTIME && IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
}
return entry_ref;
@@ -2172,6 +2173,19 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ {
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+ }
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 9dcb24db975..f5b8c7b039c 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -69,7 +69,7 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
backend_has_iostats = true;
pgstat_report_fixed = true;
@@ -89,7 +89,7 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
backend_has_iostats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 53dbf2a514b..b69a1e26f7d 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -82,7 +82,7 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
have_iostats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 1d16cde1889..36231ee874b 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -226,7 +226,7 @@ get_slru_entry(int slru_idx)
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index f0260e6e412..3bb43362e51 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2782,6 +2782,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c4f92fcdac8..6ce5a250170 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -669,6 +669,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 1651f16f966..e0f222695bf 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -816,6 +816,7 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index b6ecb0e769f..3a2ae6c41cd 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
--
2.34.1
[text/x-diff] v5-0003-Remove-useless-calls-to-flush-some-stats.patch (7.6K, ../../aYDb7g5hXnaP+5BU@ip-10-97-1-34.eu-west-3.compute.internal/4-v5-0003-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From 73aecb37f8d6f9551085a91bc731298d753d6525 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v5 3/4] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.9% src/backend/replication/
22.8% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 24d7ef795cb..1c5ffcab3e0 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -572,16 +572,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a0e6a3d200c..74102def9c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1825,7 +1821,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1846,7 +1841,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1957,8 +1951,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1967,15 +1960,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2878,8 +2862,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2974,9 +2956,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2984,7 +2963,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2995,21 +2973,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..c058a5f9b1f 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index d7e62e4d488..dda872f7074 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v5-0004-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch (20.9K, ../../aYDb7g5hXnaP+5BU@ip-10-97-1-34.eu-west-3.compute.internal/5-v5-0004-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch)
download | inline diff:
From 16df908878b6ab3e3295785e0298b024da457ec7 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v5 4/4] Add FLUSH_MIXED support and implement it for RELATION
stats
This commit extends the non transactional stats infrastructure to support statistics
kinds with mixed transaction behavior: some fields are transactional (e.g., tuple
inserts/updates/deletes) while others are non transactional (e.g., sequential scans
blocks read, ...).
It introduces FLUSH_MIXED as a third flush mode type, alongside FLUSH_ANYTIME
and FLUSH_AT_TXN_BOUNDARY. For FLUSH_MIXED kinds, a new flush_anytime_cb callback
enables partial flushing of only the non transactional fields during running
transactions.
Some tests are also added.
Implementation details:
- Add FLUSH_MIXED to PgStat_FlushMode enum
- Add flush_anytime_cb to PgStat_KindInfo for partial flushing callback
- Update pgstat_flush_pending_entries() to call flush_anytime_cb for
FLUSH_MIXED entries when in anytime_only mode
- Keep FLUSH_MIXED entries in the pending list after partial flush, as
transactional fields still need to be flushed at transaction boundary
- Add pgstat_report_mixed_anytime, a new global variable to track when the
ANYTIME_STATS_UPDATE_TIMEOUT needs to be enabled for FLUSH_MIXED mode
RELATION stats are making use of FLUSH_MIXED:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_MIXED
- Implement pgstat_relation_flush_anytime_cb() to flush only read related
stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
blocks_hit
- Clear these fields after flushing to prevent double counting when
pgstat_relation_flush_cb() runs at transaction commit
- Transactional stats (tuples_inserted, tuples_updated, tuples_deleted,
live_tuples, dead_tuples) remain pending until transaction boundary
The DATABASE kind is also changed from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME, so
that some stats inherited from relations stats are also visible while the transaction
is in progress.
Remark:
We could also imagine adding a new flush_anytime_static_cb() callback for
future FLUSH_MIXED fixed amount stats.
---
doc/src/sgml/monitoring.sgml | 29 +++++++++
src/backend/utils/activity/pgstat.c | 44 ++++++++++---
src/backend/utils/activity/pgstat_relation.c | 67 ++++++++++++++++++++
src/include/pgstat.h | 10 +++
src/include/utils/pgstat_internal.h | 11 ++++
src/test/isolation/expected/stats.out | 40 ++++++++++++
src/test/isolation/expected/stats_1.out | 40 ++++++++++++
src/test/isolation/specs/stats.spec | 17 ++++-
8 files changed, 247 insertions(+), 11 deletions(-)
14.3% doc/src/sgml/
43.1% src/backend/utils/activity/
4.9% src/include/utils/
4.6% src/include/
27.5% src/test/isolation/expected/
5.3% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..581d6ea7811 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index dd174129403..d54ccb95c6a 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -219,6 +219,12 @@ PgStat_LocalState pgStatLocal;
*/
bool pgstat_report_fixed = false;
+/*
+ * Track pending reports for mixed anytime stats, used by
+ * pgstat_report_anytime_stat().
+ */
+bool pgstat_report_mixed_anytime = false;
+
/* ----------
* Local data
*
@@ -289,7 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -307,7 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_MIXED,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -315,6 +321,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.pending_size = sizeof(PgStat_TableStatus),
.flush_pending_cb = pgstat_relation_flush_cb,
+ .flush_anytime_cb = pgstat_relation_flush_anytime_cb,
.delete_pending_cb = pgstat_relation_delete_pending_cb,
.reset_timestamp_cb = pgstat_relation_reset_timestamp_cb,
},
@@ -1303,8 +1310,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
dlist_push_tail(&pgStatPending, &entry_ref->pending_node);
/* Schedule next anytime stats update timeout */
- if (kind_info->flush_mode == FLUSH_ANYTIME && IsUnderPostmaster &&
- !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ if ((kind_info->flush_mode == FLUSH_ANYTIME || pgstat_report_mixed_anytime) &&
+ IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
}
@@ -1353,10 +1360,11 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME and FLUSH_MIXED entries,
+ * using flush_anytime_cb for FLUSH_MIXED.
* This is safe to call inside transactions.
*
- * If anytime_only is false, flushes all entries.
+ * If anytime_only is false, flushes all entries using flush_pending_cb.
*/
static bool
pgstat_flush_pending_entries(bool nowait, bool anytime_only)
@@ -1384,6 +1392,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1403,8 +1412,21 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
continue;
}
- /* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ /* flush the stats (with the appropriate callback), if possible */
+ if (anytime_only &&
+ kind_info->flush_mode == FLUSH_MIXED &&
+ kind_info->flush_anytime_cb != NULL)
+ {
+ /* Partial flush of non-transactional fields only */
+ did_flush = kind_info->flush_anytime_cb(entry_ref, nowait);
+ is_partial_flush = true;
+ }
+ else
+ {
+ /* Full flush */
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ is_partial_flush = false;
+ }
Assert(did_flush || nowait);
@@ -1414,8 +1436,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
@@ -2201,6 +2223,8 @@ pgstat_report_anytime_stat(bool force)
/* Flush stats outside of transaction boundary */
pgstat_flush_pending_entries(nowait, true);
pgstat_flush_fixed_stats(nowait, true);
+
+ pgstat_report_mixed_anytime = false;
}
/*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index feae2ae5f44..ae8778d8e39 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -887,6 +887,73 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+/*
+ * Flush only non-transactional relation stats.
+ *
+ * This is called periodically during running transactions to make some
+ * statistics visible without waiting for the transaction to finish.
+ *
+ * Transactional stats (inserts/updates/deletes and their effects on live/dead
+ * tuple counts) remain in pending until the transaction ends, at which point
+ * pgstat_relation_flush_cb() will flush them.
+ *
+ * If nowait is true and the lock could not be immediately acquired, returns
+ * false without flushing the entry. Otherwise returns true.
+ */
+bool
+pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ Oid dboid;
+ PgStat_TableStatus *lstats; /* pending stats entry */
+ PgStatShared_Relation *shtabstats;
+ PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
+ PgStat_StatDBEntry *dbentry; /* pending database entry */
+
+ dboid = entry_ref->shared_entry->key.dboid;
+ lstats = (PgStat_TableStatus *) entry_ref->pending;
+ shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ /* Add only the non-transactional values to the shared entry */
+ tabentry = &shtabstats->stats;
+
+ tabentry->numscans += lstats->counts.numscans;
+ if (lstats->counts.numscans)
+ {
+ TimestampTz t = GetCurrentTimestamp();
+
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+ tabentry->tuples_returned += lstats->counts.tuples_returned;
+ tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+ tabentry->blocks_fetched += lstats->counts.blocks_fetched;
+ tabentry->blocks_hit += lstats->counts.blocks_hit;
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ dbentry->tuples_returned += lstats->counts.tuples_returned;
+ dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ dbentry->blocks_fetched += lstats->counts.blocks_fetched;
+ dbentry->blocks_hit += lstats->counts.blocks_hit;
+
+ /*
+ * Clear the flushed fields from pending stats to prevent double-counting
+ * when pgstat_relation_flush_cb() runs at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
+}
+
void
pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref)
{
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e0f222695bf..66cc7745498 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -38,6 +38,9 @@
/* Minimum interval non-forced stats flushes */
#define PGSTAT_MIN_INTERVAL 1000
+/* Track if mixed anytime stats need to be flushed */
+extern PGDLLIMPORT bool pgstat_report_mixed_anytime;
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -693,36 +696,43 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.numscans++; \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.tuples_returned++; \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.tuples_fetched++; \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.numscans++; \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.tuples_returned += (n); \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.blocks_fetched++; \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.blocks_hit++; \
} while (0)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index a9190078d0e..db4d86cf31c 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -236,6 +236,8 @@ typedef enum PgStat_FlushMode
* transaction boundary */
FLUSH_ANYTIME, /* All fields can be flushed anytime,
* including within transactions */
+ FLUSH_MIXED, /* Mix of fields that can be flushed anytime
+ * or only at transaction boundary */
} PgStat_FlushMode;
/*
@@ -271,6 +273,12 @@ typedef struct PgStat_KindInfo
*/
PgStat_FlushMode flush_mode;
+ /*
+ * For FLUSH_MIXED kinds: callback to flush only some fields. If NULL for
+ * a MIXED kind, treated as FLUSH_AT_TXN_BOUNDARY.
+ */
+ bool (*flush_anytime_cb) (PgStat_EntryRef *entry_ref, bool nowait);
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -784,6 +792,7 @@ extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -883,6 +892,8 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 obji
*/
extern PGDLLIMPORT bool pgstat_report_fixed;
+/* Track if mixed anytime stats need to be flushed */
+
/* Backend-local stats state */
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..6d62b30e4a7 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..2fade10e817 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 0| 0| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..a4084efda49 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -435,6 +441,15 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-04 15:57 ` Zsolt Parragi <zsolt.parragi@percona.com>
2026-02-04 20:15 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Zsolt Parragi @ 2026-02-04 15:57 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org
Hello
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.numscans++; \
Shouldn't these pgstat_report_mixed_anytime changes go inside the if
statement in all macros?
+/* Track if mixed anytime stats need to be flushed */
+
/* Backend-local stats state */
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
This seems to be a leftover comment
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 15:57 ` Re: Flush some statistics within running transactions Zsolt Parragi <zsolt.parragi@percona.com>
@ 2026-02-04 20:15 ` Sami Imseih <samimseih@gmail.com>
0 siblings, 0 replies; 71+ messages in thread
From: Sami Imseih @ 2026-02-04 20:15 UTC (permalink / raw)
To: Zsolt Parragi <zsolt.parragi@percona.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org
> do { \
> + pgstat_report_mixed_anytime = true; \
> if (pgstat_should_count_relation(rel)) \
> (rel)->pgstat_info->counts.numscans++; \
>
> Shouldn't these pgstat_report_mixed_anytime changes go inside the if
> statement in all macros?
I think you are correct here, and there is an even more fundamental issue.
since
``
#define pgstat_should_count_relation(rel) \
(likely((rel)->pgstat_info != NULL) ? true : \
```
could return if there is already a pgstat_info, we may never actually
enable the timeout.
so, I think we should:
1/
remove the scheduling of the timeout from pgstat_prep_pending_entry
@@ -1308,11 +1308,6 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid
dboid, uint64 objid, bool *creat
entry_ref->pending =
MemoryContextAllocZero(pgStatPendingContext, entrysize);
dlist_push_tail(&pgStatPending, &entry_ref->pending_node);
-
- /* Schedule next anytime stats update timeout */
- if ((kind_info->flush_mode == FLUSH_ANYTIME ||
pgstat_report_mixed_anytime) &&
- IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
-
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
pgstat_flush_interval);
}
2/
Create a routine to schedule the timeout:
+void
+ScheduleAnyTimeUpdate(void)
+{
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
pgstat_flush_interval);
+
+ pgstat_report_mixed_anytime = true;
+}
This will also set pgstat_report_mixed_anytime to true. In my earlier
comments, I mentioned
having this as a public routine will be needed for extensions that
register a custom kind as well.
3/
All the count routines that wish to schedule any ANYTIME update because
their kind allows it can do so with ScheduleAnyTimeUpdate(). In the relation
stats, this can happen after it is checked that the relation should
track counts.
+#define pgstat_count_heap_scan(rel) \
+ do { \
+ if (pgstat_should_count_relation(rel)) \
+ { \
+ (rel)->pgstat_info->counts.numscans++; \
+ ScheduleAnyTimeUpdate(); \
+ } \
+ } while (0)
Also, it would be good to check if we have anytime flushes of either a mixed or
a fixed kind. Not strictly necessary, but I think it's better to avoid
needlessly entering
the flush routines.
@@ -2220,11 +2215,27 @@ pgstat_report_anytime_stat(bool force)
pgstat_assert_is_up();
/* Flush stats outside of transaction boundary */
- pgstat_flush_pending_entries(nowait, true);
- pgstat_flush_fixed_stats(nowait, true);
+ if (pgstat_report_mixed_anytime)
+ pgstat_flush_pending_entries(nowait, true);
+
+ if (pgstat_report_mixed_anytime)
+ pgstat_flush_fixed_stats(nowait, true);
pgstat_report_mixed_anytime = false;
+ pgstat_report_fixed = false;
}
I think we could benefit from a test that st track_counts to OFF inside
a transaction and re-enables it, to make sure the pgstat_should_count_relation
work is done correctly.
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-04 16:19 ` Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-02-04 16:19 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> In v5 attached I changed the design so that we don't re-enable the timeout
Thanks!
> I do think we still need it. Indeed in 0004 that helps distinguish between
> anytime flush or mixed flush (with the help of the new pgstat_report_mixed_anytime
> global variable) in pgstat_prep_pending_entry().
Will address below in the comments.
v5-0001:
1/
```
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
```
FLUSH_ANYTIME does not have any impact on Archiver. It does not have
a .flush_static_cb registered and entries are flushed directly to
shared memory via `pgstat_report_archiver` inside the `pgarch_ArchiverCopyLoop`
loop.
FLUSH_ANYTIME needs to be described a bit more that it only applies to
kinds that have flush callbacks.
2/
I suggest simplifying the code comment to this:
```
/*
- * Some stats have to be updated only at transaction boundaries (such as
- * tuples_inserted updated, deleted), so it's very important to set the
- * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default).
+ * The mode of when to flush stats. See PgStat_FlushMode for
more details.
*/
PgStat_FlushMode flush_mode;
```
PgStat_FlushMode has sufficient description.
3/
we have this pattern:
```
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
pgstat_flush_interval);
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
```
to report fixed stats. I think it will be good to turn this into a public
routine, i.e. pgstat_report_fixed_entry, that can also be used by extensions
that register custom kinds with ANYTIME flush.
extensions using variable-numbered will get the timeout enabled
via pgstat_prep_pending_entry, per v5-0004.
We may want to add a test in test_custom_stats to ensure
this ANYTIME mechanism can work with custom kinds. What do you think?
v5-0002:
Overall, this GUC seems like a good idea. 10 second default and 1
second minimum are fine.
The patch overall looks solid, and I could not find any issues.
I do have a suggestion about the documentation:
```
</term>
<listitem>
<para>
- Sets the interval at which non-transactional statistics are
made visible
- during running transactions. Non-transactional statistics include, for
- example, WAL activity and I/O operations.
- They become visible at that interval in monitoring views such as
- <link linkend="monitoring-pg-stat-io-view">
<structname>pg_stat_io</structname></link>
- and <link linkend="monitoring-pg-stat-wal-view">
<structname>pg_stat_wal</structname></link>
- during running transactions.
- If this value is specified without units, it is taken as milliseconds.
- The default is 10 seconds (<literal>10s</literal>), which is probably
- about the smallest value you would want in practice for long running
- transactions.
+ Sets the interval at which statistics that can be updated
while a transaction is still running
+ are made visible. These include, for example, WAL activity
and I/O operations.
+ Such statistics are refreshed at the specified interval and
can be observed during active
+ transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>
and
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>.
+ Other statistics are only made visible at transaction end and
are not affected by this setting.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is
generally the smallest practical
+ value for long-running transactions.
</para>
<note>
- <para>
- This parameter does not affect transactional statistics such as
- <structname>pg_stat_all_tables</structname> columns (like
- <structfield>n_tup_ins</structfield>,
<structfield>n_tup_upd</structfield>,
- <structfield>n_tup_del</structfield>), which are always
flushed at transaction
- boundaries to maintain consistency.
- </para>
+ <para> This parameter does not affect statistics that are
only reported at transaction end,
+ such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>,
<structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics
are always flushed at the end of
+ a transaction. </para>
</note>
</listitem>
</varlistentry>
```
specifically, I want to avoid using "non-transactional", "transaction
boundary" terms, as
they may be confusing.
v5-0003: looks straightforward. I have no comments.
v5-0004:
/1
+ FLUSH_MIXED, /* Mix of fields that
can be flushed anytime
+ * or
only at transaction boundary */
I still don't think this is needed. I fail to see what value it adds.
When we set pgstat_report_mixed_anytime inside the pgstat_count functions,
which make it clear which of the fields are anytime fields.
2/
This is an oversight comment, it seems.
```
extern PGDLLIMPORT bool pgstat_report_fixed;
+/* Track if mixed anytime stats need to be flushed */
+
/* Backend-local stats state */
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
```
3/
```
+ tabentry->numscans += lstats->counts.numscans;
+ if (lstats->counts.numscans)
+ {
+ TimestampTz t = GetCurrentTimestamp();
```
Considering the 10 second default, GetCurrentTimestamp() should not be
an issue here.
The doc does need to be updated. From:
"The time of the last sequential scan on this table, based on the most
recent transaction stop time"
To this?
"The approximate time of the last sequential scan on this table,
updated at least every STATS_FLUSH_INTERVAL"
same for `last_index_scan`
4/
nit: /non partial flush/non-partial flush/
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-04 16:26 ` Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-02-04 16:26 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> Will address below in the comments.
Sorry there was one more comment I forgot about.
in v5-0004, instead of `flush_anytime_cb`, can't we
just pass the anytime flag to the `flush_pending_cb` and
take care of what needs to be flushed there?
It will be up to the author of the kind if they want to further
split work in multiple functions, but we could just have a
single callback that is ANYTIME aware.
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-06 11:09 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-06 11:09 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Wed, Feb 04, 2026 at 10:26:35AM -0600, Sami Imseih wrote:
> > Will address below in the comments.
>
> Sorry there was one more comment I forgot about.
>
> in v5-0004, instead of `flush_anytime_cb`, can't we
> just pass the anytime flag to the `flush_pending_cb` and
> take care of what needs to be flushed there?
>
> It will be up to the author of the kind if they want to further
> split work in multiple functions, but we could just have a
> single callback that is ANYTIME aware.
Thank you Sami and Zsolt for the reviews!
PFA attached v6, addressing the reviews comments.
Main changes are:
0001:
- Adds a pgstat_schedule_anytime_update() macro to schedule the next anytime flush
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducing
a new callback) in 0005.
0002:
Add some tests for anytime flush on custom stats. It adds a few functions:
we could/may want to avoid some duplicate code and update the version to 1.1
though. For the former, I don't have a strong opinion and for the later I'm not
sure (for example 07ff701dbd53 did not bump the version).
0005:
Get rid of the FLUSH_MIXED mode, the new anytime callback and the
pgstat_report_mixed_anytime variable introduced in v5. Instead v6:
- change RELATION and DATABASE kinds from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats
Remark:
I did not add any additional checks in pgstat_report_anytime_stat(), since we
enter it for good reason: at least one the flush has to be triggered. I'm not sure
it's worth to add more check to try to discard one of them. The flush functions
return early if they don't have anything to flush anyway. So the gain would be
to try to avoid one function call at pgstat_flush_interval interval, but we would
need to maitain the variable(s) and avoid possible race conditions with
pgstat_report_stat().
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v6-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (41.9K, ../../aYXL6XmVABORjGF5@ip-10-97-1-34.eu-west-3.compute.internal/2-v6-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From b2e59b407ca3fd60f5f917c9ec5f93047d0ac649 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v6 1/5] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces
spikes when flushed.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducin
a new callback).
- Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush,
relying on PGSTAT_MIN_INTERVAL
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 6 +
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/replication/walsender.c | 8 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 112 ++++++++++++++----
src/backend/utils/activity/pgstat_backend.c | 13 +-
src/backend/utils/activity/pgstat_bgwriter.c | 2 +-
.../utils/activity/pgstat_checkpointer.c | 2 +-
src/backend/utils/activity/pgstat_database.c | 2 +-
src/backend/utils/activity/pgstat_function.c | 4 +-
src/backend/utils/activity/pgstat_io.c | 10 +-
src/backend/utils/activity/pgstat_relation.c | 12 +-
src/backend/utils/activity/pgstat_slru.c | 6 +-
.../utils/activity/pgstat_subscription.c | 4 +-
src/backend/utils/activity/pgstat_wal.c | 10 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 16 +++
src/include/utils/pgstat_internal.h | 52 ++++++--
src/include/utils/timeout.h | 1 +
.../test_custom_stats/test_custom_var_stats.c | 4 +-
src/tools/pgindent/typedefs.list | 1 +
28 files changed, 264 insertions(+), 66 deletions(-)
10.8% src/backend/postmaster/
6.0% src/backend/replication/
50.7% src/backend/utils/activity/
6.0% src/backend/
19.3% src/include/utils/
5.6% src/include/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..d01b11c7470 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0956bd39a85..059c601c3b8 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..e11c4b099c8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..4954fe425b7 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 2d8f57099fd..9f8ef8159d1 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -249,7 +251,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -271,6 +273,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 23e79a32345..ded0f250288 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -106,7 +108,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -116,6 +118,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 10e64a7d1f4..11b7c114d3b 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2cde8ebc729..a7214d0dc6f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
@@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 02e9aaa6bca..c7bc409b06f 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3509,6 +3509,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..411b65aae3e 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -112,6 +112,7 @@
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +123,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +784,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
@@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
@@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait, anytime_only);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
+
+/*
+ * Timeout handler for flushing non-transactional stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 1350f5f62f1..b0bd220882d 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
* if some statistics could not be flushed due to lock contention.
*/
bool
-pgstat_flush_backend(bool nowait, bits32 flags)
+pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only)
{
PgStat_EntryRef *entry_ref;
bool has_pending_data = false;
@@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
* If some stats could not be flushed due to lock contention, return true.
*/
bool
-pgstat_backend_flush_cb(bool nowait)
+pgstat_backend_flush_cb(bool nowait, bool anytime_only)
{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
+ return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only);
}
/*
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index ed2fd801189..1c5f0c3ec40 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -61,7 +61,7 @@ pgstat_report_bgwriter(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 1f70194b7a7..2d89a082464 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -68,7 +68,7 @@ pgstat_report_checkpointer(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index d7f6d4c5ee6..61094f96a6c 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -425,7 +425,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index e6b84283c6c..5ba4958382f 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
+ Assert(!anytime_only);
+
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..7cd32900236 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void)
* Simpler wrapper of pgstat_io_flush_cb()
*/
void
-pgstat_flush_io(bool nowait)
+pgstat_flush_io(bool nowait, bool anytime_only)
{
- (void) pgstat_io_flush_cb(nowait);
+ (void) pgstat_io_flush_cb(nowait, anytime_only);
}
/*
@@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait)
* acquired. Otherwise, return false.
*/
bool
-pgstat_io_flush_cb(bool nowait)
+pgstat_io_flush_cb(bool nowait, bool anytime_only)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..04d21483d93 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* is done -- which will likely vacuum many relations -- or until the
* VACUUM command has processed all tables and committed.
*/
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel,
pgstat_unlock_entry(entry_ref);
/* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
* entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
+ Assert(!anytime_only);
+
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..bf8a4d58673 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name)
* acquired. Otherwise return false.
*/
bool
-pgstat_slru_flush_cb(bool nowait)
+pgstat_slru_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
int i;
@@ -223,6 +224,9 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 500b1899188..c4614817966 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
+ Assert(!anytime_only);
+
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..eb5f8f46925 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -51,12 +51,12 @@ pgstat_report_wal(bool force)
nowait = !force;
/* flush wal stats */
- (void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
+ (void) pgstat_wal_flush_cb(nowait, true);
+ pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
/* flush IO stats */
- pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -88,7 +88,7 @@ pgstat_wal_have_pending(void)
* acquired. Otherwise return false.
*/
bool
-pgstat_wal_flush_cb(bool nowait)
+pgstat_wal_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
WalUsage wal_usage_diff = {0};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..f45365f47f7 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -765,6 +766,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..8aeb9628871 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..b340a680614 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,8 +536,21 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
+/*
+ * Schedule the next anytime stats update timeout.
+ *
+ * This should be called whenever accumulating statistics that support
+ * FLUSH_ANYTIME flushing mode.
+ */
+#define pgstat_schedule_anytime_update() \
+ do { \
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ } while (0)
+
extern void pgstat_reset_counters(void);
extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_reset_of_kind(PgStat_Kind kind);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..607f4255268 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * The mode of when to flush stats. See PgStat_FlushMode for more details.
+ *
+ * This member only has meaning for statistics kinds that accumulate
+ * pending stats and use flush callbacks. For kinds that write directly to
+ * shared memory (e.g., archiver, bgwriter, checkpointer), this member has
+ * no effect.
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo
*
* "pgstat_report_fixed" needs to be set to trigger the flush of pending
* stats.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_static_cb) (bool nowait);
+ bool (*flush_static_cb) (bool nowait, bool anytime_only);
/*
* For fixed-numbered statistics: Reset All.
@@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
@@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
-extern bool pgstat_flush_backend(bool nowait, bits32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
+extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only);
+extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_io.c
*/
-extern void pgstat_flush_io(bool nowait);
+extern void pgstat_flush_io(bool nowait, bool anytime_only);
-extern bool pgstat_io_flush_cb(bool nowait);
+extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
* Functions in pgstat_slru.c
*/
-extern bool pgstat_slru_flush_cb(bool nowait);
+extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void);
*/
extern void pgstat_wal_init_backend_cb(void);
-extern bool pgstat_wal_flush_cb(bool nowait);
+extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
@@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 64a8fe63cce..bc0b5d6e0eb 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait);
+ bool nowait, bool anytime_only);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -150,7 +150,7 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9f5ee8fd482..860f835c088 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v6-0002-Add-anytime-flush-tests-for-custom-stats.patch (9.3K, ../../aYXL6XmVABORjGF5@ip-10-97-1-34.eu-west-3.compute.internal/3-v6-0002-Add-anytime-flush-tests-for-custom-stats.patch)
download | inline diff:
From c381419eb6d5018c3fe5476ac6edae8673ac5343 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 5 Feb 2026 05:54:34 +0000
Subject: [PATCH v6 2/5] Add anytime flush tests for custom stats
---
.../test_custom_stats/t/001_custom_stats.pl | 41 ++++++++++++
.../test_custom_fixed_stats--1.0.sql | 10 +++
.../test_custom_fixed_stats.c | 66 +++++++++++++++++++
.../test_custom_var_stats--1.0.sql | 5 ++
.../test_custom_stats/test_custom_var_stats.c | 27 ++++++++
5 files changed, 149 insertions(+)
31.4% src/test/modules/test_custom_stats/t/
68.5% src/test/modules/test_custom_stats/
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 9e6a7a38577..36d9fc3fde1 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -156,5 +156,46 @@ $result = $node->safe_psql('postgres',
);
is($result, "0", "report of fixed-sized after manual reset");
+# Test FLUSH_ANYTIME mechanism with custom fixed stats
+# This verifies that custom stats can be flushed during a transaction
+
+# Reset stats first
+$node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+my $anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
+ -- Force anytime flush (inside transaction!)
+ select pg_stat_force_anytime_flush();
+ -- Check
+ select 'anytime:'||numcalls from test_custom_stats_fixed_report();
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^anytime:2/m,
+ "anytime fixed stats flushed during transaction");
+
+# Test FLUSH_ANYTIME mechanism with custom variable stats
+# This verifies that custom stats can be flushed during a transaction
+
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+$anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_var_anytime_update('entry2');
+ select test_custom_stats_var_anytime_update('entry2');
+ -- Force anytime flush (inside transaction!)
+ select pg_stat_force_anytime_flush();
+ -- Check
+ select * from test_custom_stats_var_report('entry2');
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^entry2|2|/m,
+ "anytime var stats flushed during transaction");
+
# Test completed successfully
done_testing();
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
index 69a93b5241f..c0a418c3ae3 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
@@ -18,3 +18,13 @@ CREATE FUNCTION test_custom_stats_fixed_reset()
RETURNS void
AS 'MODULE_PATHNAME', 'test_custom_stats_fixed_reset'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_fixed_anytime_update()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION pg_stat_force_anytime_flush()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
index 908bd18a7c7..6b3bc3257ab 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
@@ -18,6 +18,7 @@
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_fixed_stats",
@@ -43,11 +44,13 @@ typedef struct PgStatShared_CustomFixedEntry
static void test_custom_stats_fixed_init_shmem_cb(void *stats);
static void test_custom_stats_fixed_reset_all_cb(TimestampTz ts);
static void test_custom_stats_fixed_snapshot_cb(void);
+static bool test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only);
static const PgStat_KindInfo custom_stats = {
.name = "test_custom_fixed_stats",
.fixed_amount = true, /* exactly one entry */
.write_to_file = true, /* persist to stats file */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.shared_size = sizeof(PgStat_StatCustomFixedEntry),
.shared_data_off = offsetof(PgStatShared_CustomFixedEntry, stats),
@@ -56,8 +59,12 @@ static const PgStat_KindInfo custom_stats = {
.init_shmem_cb = test_custom_stats_fixed_init_shmem_cb,
.reset_all_cb = test_custom_stats_fixed_reset_all_cb,
.snapshot_cb = test_custom_stats_fixed_snapshot_cb,
+ .flush_static_cb = test_custom_stats_fixed_flush_cb,
};
+/* Pending statistics */
+static PgStat_StatCustomFixedEntry PendingCustomStats = {0};
+
/*
* Kind ID for test_custom_fixed_stats.
*/
@@ -141,6 +148,38 @@ test_custom_stats_fixed_snapshot_cb(void)
#undef FIXED_COMP
}
+/*
+ * test_custom_stats_fixed_flush_cb
+ * Flush pending stats to shared memory
+ */
+static bool
+test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only)
+{
+ PgStatShared_CustomFixedEntry *stats_shmem;
+
+ /* Nothing to flush if no calls were made */
+ if (PendingCustomStats.numcalls == 0)
+ return false;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_TEST_CUSTOM_FIXED_STATS);
+
+ if (nowait && !LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true; /* failed to flush */
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numcalls += PendingCustomStats.numcalls;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ LWLockRelease(&stats_shmem->lock);
+
+ /* Reset pending stats */
+ PendingCustomStats.numcalls = 0;
+
+ return false; /* successfully flushed */
+}
+
/*--------------------------------------------------------------------------
* SQL-callable functions
*--------------------------------------------------------------------------
@@ -222,3 +261,30 @@ test_custom_stats_fixed_report(PG_FUNCTION_ARGS)
/* Return as tuple */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/*
+ * test_custom_stats_fixed_anytime_update
+ * Increment call counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_fixed_anytime_update);
+Datum
+test_custom_stats_fixed_anytime_update(PG_FUNCTION_ARGS)
+{
+ /* Accumulate in pending stats */
+ PendingCustomStats.numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+ pgstat_report_fixed = true;
+
+ PG_RETURN_VOID();
+}
+
+/* Helper function for testing ANYTIME flush */
+PG_FUNCTION_INFO_V1(pg_stat_force_anytime_flush);
+Datum
+pg_stat_force_anytime_flush(PG_FUNCTION_ARGS)
+{
+ pgstat_report_anytime_stat(true);
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
index 5ed8cfc2dcf..ed66d38981e 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
@@ -24,3 +24,8 @@ CREATE FUNCTION test_custom_stats_var_report(INOUT name TEXT,
RETURNS SETOF record
AS 'MODULE_PATHNAME', 'test_custom_stats_var_report'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_var_anytime_update(IN name TEXT)
+RETURNS void
+AS 'MODULE_PATHNAME', 'test_custom_stats_var_anytime_update'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index bc0b5d6e0eb..207e841911b 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -17,6 +17,7 @@
#include "storage/dsm_registry.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_var_stats",
@@ -107,6 +108,7 @@ static const PgStat_KindInfo custom_stats = {
.name = "test_custom_var_stats",
.fixed_amount = false, /* variable number of entries */
.write_to_file = true, /* persist across restarts */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.track_entry_count = true, /* count active entries */
.accessed_across_databases = true, /* global statistics */
.shared_size = sizeof(PgStatShared_CustomVarEntry),
@@ -689,3 +691,28 @@ test_custom_stats_var_report(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * test_custom_stats_var_anytime_update
+ * Increment custom statistic counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_var_anytime_update);
+Datum
+test_custom_stats_var_anytime_update(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_EntryRef *entry_ref;
+ PgStat_StatCustomVarEntry *pending_entry;
+
+ /* Get pending entry in local memory */
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, InvalidOid,
+ PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name), NULL);
+
+ pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
+ pending_entry->numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+
+ PG_RETURN_VOID();
+}
--
2.34.1
[text/x-diff] v6-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch (7.6K, ../../aYXL6XmVABORjGF5@ip-10-97-1-34.eu-west-3.compute.internal/4-v6-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch)
download | inline diff:
From 69f3e4f063a956f429ee0dbdbd213dcad8242613 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v6 3/5] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 34 +++++++++++++++++++
src/backend/utils/activity/pgstat.c | 16 +++++++++
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/pgstat.h | 6 ++--
src/include/utils/guc_hooks.h | 1 +
6 files changed, 64 insertions(+), 4 deletions(-)
59.1% doc/src/sgml/
14.2% src/backend/utils/activity/
14.5% src/backend/utils/misc/
12.0% src/include/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f1af1505cf3..20666679f90 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8871,6 +8871,40 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which statistics that can be updated while a
+ transaction is still running are made visible. These include, for example,
+ WAL activity and I/O operations.
+ Such statistics are refreshed at the specified interval and can be observed
+ during active transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>
+ and
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>.
+ Other statistics are only made visible at transaction end and are not
+ affected by this setting.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is generally
+ the smallest practical value for long-running transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect statistics that are only reported at
+ transaction end, such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics are always
+ flushed at the end of a transaction.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 411b65aae3e..79eb59b5625 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -123,6 +123,8 @@
* ----------
*/
+/* minimum interval non-forced stats flushes.*/
+#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -203,6 +205,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -2164,6 +2167,19 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ {
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+ }
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index c1f1603cd39..fc0e4259b36 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2789,6 +2789,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1ae594af843..3f998a0bea0 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -673,6 +673,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b340a680614..ef856dbf55b 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,9 +35,6 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
-/* Minimum interval non-forced stats flushes */
-#define PGSTAT_MIN_INTERVAL 1000
-
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -548,7 +545,7 @@ extern void pgstat_force_next_flush(void);
#define pgstat_schedule_anytime_update() \
do { \
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval); \
} while (0)
extern void pgstat_reset_counters(void);
@@ -828,6 +825,7 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index b6ecb0e769f..3a2ae6c41cd 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
--
2.34.1
[text/x-diff] v6-0004-Remove-useless-calls-to-flush-some-stats.patch (7.6K, ../../aYXL6XmVABORjGF5@ip-10-97-1-34.eu-west-3.compute.internal/5-v6-0004-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From de9afb385c171f3e0f8de2a0dbe4f72143216686 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v6 4/5] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.4% src/test/recovery/t/
3.5% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..c058a5f9b1f 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..bceec2adede 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v6-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch (26.2K, ../../aYXL6XmVABORjGF5@ip-10-97-1-34.eu-west-3.compute.internal/6-v6-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch)
download | inline diff:
From 5a57a7034acf37b11f9020e415c801963270bd33 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v6 5/5] Change RELATION and DATABASE stats to anytime flush
This commit allows mixing fields with different transaction behavior within
the same RELATION or DATABASE statistics kind: some fields are transactional
(e.g., tuple inserts/updates/deletes) while others are non-transactional
(e.g., sequential scans, blocks read).
It modifies the relation flush callback to handle the anytime_only parameter
introduced in commit <nnnn>.
Implementation details:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Change DATABASE from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats. When set to true, it clears flushed fields from pending stats to
prevent double-counting at transaction boundary
DATABASE stats inherit the anytime flush behavior so that relation-derived
stats (tuples_returned, tuples_fetched, blocks_fetched, blocks_hit) are
visible while transactions are in progress.
Tests are added to verify the anytime flush behavior for mixed fields.
---
doc/src/sgml/monitoring.sgml | 37 ++++++-
src/backend/utils/activity/pgstat.c | 17 ++--
src/backend/utils/activity/pgstat_relation.c | 86 ++++++++++++----
src/include/pgstat.h | 27 ++++-
src/test/isolation/expected/stats.out | 102 +++++++++++++++++++
src/test/isolation/expected/stats_1.out | 102 +++++++++++++++++++
src/test/isolation/specs/stats.spec | 27 ++++-
7 files changed, 368 insertions(+), 30 deletions(-)
13.0% doc/src/sgml/
25.9% src/backend/utils/activity/
6.0% src/include/
49.8% src/test/isolation/expected/
5.1% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..aa7bd2e2e2a 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -3956,8 +3969,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_seq_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last sequential scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last sequential scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -3984,8 +3997,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_idx_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last index scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last index scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 79eb59b5625..9234185de64 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -291,7 +291,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -309,7 +309,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -1344,10 +1344,12 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
dlist_delete(&entry_ref->pending_node);
}
+
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries. For entries
+ * that support it, the callback may flush only non-transactional fields.
* This is safe to call inside transactions.
*
* If anytime_only is false, flushes all entries.
@@ -1378,6 +1380,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1397,19 +1400,21 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
continue;
}
- /* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
+ /* Partial flush only happens in anytime mode for FLUSH_ANYTIME stats */
+ is_partial_flush = (anytime_only && kind_info->flush_mode == FLUSH_ANYTIME);
+
/* determine next entry, before deleting the pending entry */
if (dlist_has_next(&pgStatPending, cur))
next = dlist_next_node(&pgStatPending, cur);
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ae2952cae89..10ab9e4cc76 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -47,7 +47,19 @@ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_lev
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
+static void flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry,
+ PgStat_TableCounts *counts, bool anytime_only);
+/*
+ * Update database statistics with non-transactional stats.
+ */
+#define UPDATE_DATABASE_ANYTIME_STATS(dbentry, counts) \
+ do { \
+ (dbentry)->tuples_returned += (counts)->tuples_returned; \
+ (dbentry)->tuples_fetched += (counts)->tuples_fetched; \
+ (dbentry)->blocks_fetched += (counts)->blocks_fetched; \
+ (dbentry)->blocks_hit += (counts)->blocks_hit; \
+ } while (0)
/*
* Copy stats between relations. This is used for things like REINDEX
@@ -789,6 +801,29 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Helper function to flush non-transactional statistics.
+ */
+static void
+flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry, PgStat_TableCounts *counts,
+ bool anytime_only)
+{
+ TimestampTz t;
+
+ tabentry->numscans += counts->numscans;
+ if (counts->numscans)
+ {
+ t = anytime_only ? GetCurrentTimestamp() : GetCurrentTransactionStopTimestamp();
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+
+ tabentry->tuples_returned += counts->tuples_returned;
+ tabentry->tuples_fetched += counts->tuples_fetched;
+ tabentry->blocks_fetched += counts->blocks_fetched;
+ tabentry->blocks_hit += counts->blocks_hit;
+}
+
/*
* Flush out pending stats for the entry
*
@@ -797,6 +832,13 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
*
* Some of the stats are copied to the corresponding pending database stats
* entry when successfully flushing.
+ *
+ * If anytime_only is true, only non-transactional fields are flushed
+ * (numscans, tuples_returned, tuples_fetched, blocks_fetched, blocks_hit).
+ * Transactional fields remain pending until transaction boundary.
+ *
+ * Some of the stats are copied to the corresponding pending database stats
+ * entry when successfully flushing.
*/
bool
pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
@@ -807,8 +849,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
- Assert(!anytime_only);
-
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
@@ -824,19 +864,36 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
- /* add the values to the shared entry. */
tabentry = &shtabstats->stats;
- tabentry->numscans += lstats->counts.numscans;
- if (lstats->counts.numscans)
+ if (anytime_only)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
- if (t > tabentry->lastscan)
- tabentry->lastscan = t;
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, true);
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
+ /*
+ * Clear the flushed fields from pending stats to prevent
+ * double-counting when we flush all fields at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
}
- tabentry->tuples_returned += lstats->counts.tuples_returned;
- tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, false);
+
tabentry->tuples_inserted += lstats->counts.tuples_inserted;
tabentry->tuples_updated += lstats->counts.tuples_updated;
tabentry->tuples_deleted += lstats->counts.tuples_deleted;
@@ -866,9 +923,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
*/
tabentry->ins_since_vacuum += lstats->counts.tuples_inserted;
- tabentry->blocks_fetched += lstats->counts.blocks_fetched;
- tabentry->blocks_hit += lstats->counts.blocks_hit;
-
/* Clamp live_tuples in case of negative delta_live_tuples */
tabentry->live_tuples = Max(tabentry->live_tuples, 0);
/* Likewise for dead_tuples */
@@ -878,13 +932,11 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
/* The entry was successfully flushed, add the same to database stats */
dbentry = pgstat_prep_database_pending(dboid);
- dbentry->tuples_returned += lstats->counts.tuples_returned;
- dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
dbentry->tuples_inserted += lstats->counts.tuples_inserted;
dbentry->tuples_updated += lstats->counts.tuples_updated;
dbentry->tuples_deleted += lstats->counts.tuples_deleted;
- dbentry->blocks_fetched += lstats->counts.blocks_fetched;
- dbentry->blocks_hit += lstats->counts.blocks_hit;
return true;
}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ef856dbf55b..06639198f28 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -21,6 +21,7 @@
#include "utils/backend_status.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/pgstat_kind.h"
#include "utils/relcache.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h" /* for backward compatibility */ /* IWYU pragma: export */
@@ -537,10 +538,11 @@ extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
/*
- * Schedule the next anytime stats update timeout.
+ * Schedule the next anytime stats update timeout and mark that we have
+ * mixed anytime stats pending.
*
* This should be called whenever accumulating statistics that support
- * FLUSH_ANYTIME flushing mode.
+ * FLUSH_ANYTIME or FLUSH_MIXED flushing modes.
*/
#define pgstat_schedule_anytime_update() \
do { \
@@ -703,37 +705,58 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned += (n); \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_hit++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..11e3e57806d 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..aef582e7582 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..47414eb6009 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -156,6 +162,8 @@ step s2_func_stats {
}
# Relation stats steps
+step s2_track_counts_on { SET track_counts = on; }
+step s2_track_counts_off { SET track_counts = off; }
step s2_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; }
step s2_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';}
@@ -435,6 +443,23 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_off
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_on
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-14 02:23 ` Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-02-14 02:23 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> PFA attached v6, addressing the reviews comments.
Thanks for the patches!
v6 is getting closer IMO. Here are some comments I have.
v6-0001 looks solid, but some minor comments:
1/
+ pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
let's also use explicit (void) cast here
2/
+ * Timeout handler for flushing non-transactional stats.
I also noticed in v6-0005, we refer to "anytime" stats as
"non-transactional". It's better to just refer to them as "anytime"
anywhere instead of "non-transactional:.
v6-0002:
+ if (nowait && !LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true; /* failed to flush */
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
Here the LWLock acquisition will deadlock if we are nowait, successfully
acquire the lock conditionally, and then we try to acquire it again. The
logic here should not try to acquire the lock twice.
+ -- Force anytime flush (inside transaction!)
+ select pg_stat_force_anytime_flush();
Not sure why we need pg_stat_force_anytime_flush.
A pg_sleep is sufficient, like below. right?
```
select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
select pg_sleep(1.5);
select 'anytime:'||numcalls from test_custom_stats_fixed_report();
```
v6-0003:
1/
Suggested doc changes:
<para>
- Sets the interval at which statistics that can be updated while a
- transaction is still running are made visible. These include,
for example,
- WAL activity and I/O operations.
+ Sets the interval at which certain statistics, which can be
updated while a
+ transaction is in progress, are made visible. These include
WAL activity
+ and I/O operations.
Such statistics are refreshed at the specified interval and
can be observed
during active transactions in monitoring views such as
- <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>
and
- <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>.
- Other statistics are only made visible at transaction end and are not
- affected by this setting.
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>.
If the value is specified without a unit, milliseconds are assumed.
The default is 10 seconds (<literal>10s</literal>), which is generally
the smallest practical value for long-running transactions.
- Other statistics are only made visible at transaction end and are not
- affected by this setting.
I removed this, because it's mentioned in the notes section later on.
2/
I don't see we have tests for other timeout based GUCs, but it would nice
to ensure that this woks correctly. Maybe as a custom_stats test where we
SET stats_flush_interval inside the transaction and make sure the stats flush
only after the new timeout has passed. Maybe?
v6-0004:
1/
NIT:
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
missing space before the equal sign.
v6-0005:
1/
/* Partial flush only happens in anytime mode for FLUSH_ANYTIME stats */
is_partial_flush = (anytime_only && kind_info->flush_mode ==
FLUSH_ANYTIME);
Will this be tue at all time? Let's imagine a Kind that flushes all the fields
ANYTIME, would we not want to delete the pending entry?
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
2/ indentation:
Some statistics are updated while a transaction is in progress
(for example,
<structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
<structfield>tup_returned</structfield> and
<structfield>tup_fetched</structfield>).
- Statistics that either do not depend on transactions or require
transactional
- consistency are updated only when the transaction ends.
Statistics that require
- transactional consistency include <structfield>xact_commit</structfield>,
- <structfield>xact_rollback</structfield>,
<structfield>tup_inserted</structfield>,
- <structfield>tup_updated</structfield> and
<structfield>tup_deleted</structfield>.
+ Statistics that either do not depend on transactions or require
transactional
+ consistency are updated only when the transaction ends.
Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>,
<structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and
<structfield>tup_deleted</structfield>.
</para>
</note>
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-16 08:27 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-16 08:27 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Fri, Feb 13, 2026 at 08:23:01PM -0600, Sami Imseih wrote:
> > PFA attached v6, addressing the reviews comments.
>
> v6 is getting closer IMO. Here are some comments I have.
Thanks!
> v6-0001 looks solid, but some minor comments:
> 1/
>
> + pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
>
> let's also use explicit (void) cast here
Yeah. The patch did not introduce this inconsistency but let's fix it in passing.
> 2/
>
> + * Timeout handler for flushing non-transactional stats.
>
> I also noticed in v6-0005, we refer to "anytime" stats as
> "non-transactional". It's better to just refer to them as "anytime"
> anywhere instead of "non-transactional:.
Done.
> v6-0002:
>
> The logic here should not try to acquire the lock twice.
Yeah, changed to match the pgstat_wal_flush_cb() logic.
> + -- Force anytime flush (inside transaction!)
> + select pg_stat_force_anytime_flush();
>
> Not sure why we need pg_stat_force_anytime_flush.
> A pg_sleep is sufficient, like below. right?
Right, done.
>
> v6-0003:
>
> 1/
> Suggested doc changes:
Done.
> 2/
> I don't see we have tests for other timeout based GUCs, but it would nice
> to ensure that this woks correctly. Maybe as a custom_stats test where we
> SET stats_flush_interval inside the transaction and make sure the stats flush
> only after the new timeout has passed. Maybe?
Not sure I follow, that's what 0002 is doing.
> v6-0004:
>
> 1/
>
> NIT:
>
> +$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
>
> +$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
>
> missing space before the equal sign.
Done.
>
> v6-0005:
>
> 1/
>
> /* Partial flush only happens in anytime mode for FLUSH_ANYTIME stats */
> is_partial_flush = (anytime_only && kind_info->flush_mode ==
> FLUSH_ANYTIME);
>
> Will this be tue at all time? Let's imagine a Kind that flushes all the fields
> ANYTIME, would we not want to delete the pending entry?
>
> + /* if successfull non-partial flush, remove entry */
> + if (did_flush && !is_partial_flush)
> pgstat_delete_pending_entry(entry_ref);
>
Right, and that's why the MIXED flush mode was useful, i.e to be able to distinguish
here. So, in the attached, instead of re-introducing the MIXED flush mode, I added
a "is_partial" bool paramater to the flush_pending_cb().
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v7-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (41.9K, ../../aZLVDJLTiFlxsWZi@ip-10-97-1-34.eu-west-3.compute.internal/2-v7-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From d11748a6d20de2de9064a5293c8251031997259b Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v7 1/5] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces
spikes when flushed.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducin
a new callback).
- Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush,
relying on PGSTAT_MIN_INTERVAL
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 6 +
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/replication/walsender.c | 8 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 112 ++++++++++++++----
src/backend/utils/activity/pgstat_backend.c | 13 +-
src/backend/utils/activity/pgstat_bgwriter.c | 2 +-
.../utils/activity/pgstat_checkpointer.c | 2 +-
src/backend/utils/activity/pgstat_database.c | 2 +-
src/backend/utils/activity/pgstat_function.c | 4 +-
src/backend/utils/activity/pgstat_io.c | 10 +-
src/backend/utils/activity/pgstat_relation.c | 12 +-
src/backend/utils/activity/pgstat_slru.c | 6 +-
.../utils/activity/pgstat_subscription.c | 4 +-
src/backend/utils/activity/pgstat_wal.c | 10 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 16 +++
src/include/utils/pgstat_internal.h | 52 ++++++--
src/include/utils/timeout.h | 1 +
.../test_custom_stats/test_custom_var_stats.c | 4 +-
src/tools/pgindent/typedefs.list | 1 +
28 files changed, 264 insertions(+), 66 deletions(-)
10.8% src/backend/postmaster/
6.0% src/backend/replication/
50.7% src/backend/utils/activity/
6.0% src/backend/
19.3% src/include/utils/
5.6% src/include/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..d01b11c7470 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0956bd39a85..059c601c3b8 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..e11c4b099c8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..4954fe425b7 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 2d8f57099fd..9f8ef8159d1 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -249,7 +251,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -271,6 +273,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 23e79a32345..ded0f250288 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -106,7 +108,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -116,6 +118,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 10e64a7d1f4..11b7c114d3b 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2cde8ebc729..a7214d0dc6f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
@@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 21de158adbb..2089de782d5 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3564,6 +3564,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..a4ff64dc5ce 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -112,6 +112,7 @@
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +123,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +784,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
@@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
@@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait, anytime_only);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
+
+/*
+ * Timeout handler for flushing anytime stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index f2f8d3ff75f..b09316d3ab3 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
* if some statistics could not be flushed due to lock contention.
*/
bool
-pgstat_flush_backend(bool nowait, bits32 flags)
+pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only)
{
PgStat_EntryRef *entry_ref;
bool has_pending_data = false;
@@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
* If some stats could not be flushed due to lock contention, return true.
*/
bool
-pgstat_backend_flush_cb(bool nowait)
+pgstat_backend_flush_cb(bool nowait, bool anytime_only)
{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
+ return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only);
}
/*
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index ed2fd801189..1c5f0c3ec40 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -61,7 +61,7 @@ pgstat_report_bgwriter(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 1f70194b7a7..2d89a082464 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -68,7 +68,7 @@ pgstat_report_checkpointer(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 933dcb5cae5..8e86df60461 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index e6b84283c6c..5ba4958382f 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
+ Assert(!anytime_only);
+
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..7cd32900236 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void)
* Simpler wrapper of pgstat_io_flush_cb()
*/
void
-pgstat_flush_io(bool nowait)
+pgstat_flush_io(bool nowait, bool anytime_only)
{
- (void) pgstat_io_flush_cb(nowait);
+ (void) pgstat_io_flush_cb(nowait, anytime_only);
}
/*
@@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait)
* acquired. Otherwise, return false.
*/
bool
-pgstat_io_flush_cb(bool nowait)
+pgstat_io_flush_cb(bool nowait, bool anytime_only)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..04d21483d93 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* is done -- which will likely vacuum many relations -- or until the
* VACUUM command has processed all tables and committed.
*/
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel,
pgstat_unlock_entry(entry_ref);
/* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
* entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
+ Assert(!anytime_only);
+
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..bf8a4d58673 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name)
* acquired. Otherwise return false.
*/
bool
-pgstat_slru_flush_cb(bool nowait)
+pgstat_slru_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
int i;
@@ -223,6 +224,9 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 500b1899188..c4614817966 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
+ Assert(!anytime_only);
+
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..2c2f3f10e10 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -51,12 +51,12 @@ pgstat_report_wal(bool force)
nowait = !force;
/* flush wal stats */
- (void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
+ (void) pgstat_wal_flush_cb(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
/* flush IO stats */
- pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -88,7 +88,7 @@ pgstat_wal_have_pending(void)
* acquired. Otherwise return false.
*/
bool
-pgstat_wal_flush_cb(bool nowait)
+pgstat_wal_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
WalUsage wal_usage_diff = {0};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..eeeac1bf39a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..84e698da214 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..b340a680614 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,8 +536,21 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
+/*
+ * Schedule the next anytime stats update timeout.
+ *
+ * This should be called whenever accumulating statistics that support
+ * FLUSH_ANYTIME flushing mode.
+ */
+#define pgstat_schedule_anytime_update() \
+ do { \
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ } while (0)
+
extern void pgstat_reset_counters(void);
extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_reset_of_kind(PgStat_Kind kind);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..607f4255268 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * The mode of when to flush stats. See PgStat_FlushMode for more details.
+ *
+ * This member only has meaning for statistics kinds that accumulate
+ * pending stats and use flush callbacks. For kinds that write directly to
+ * shared memory (e.g., archiver, bgwriter, checkpointer), this member has
+ * no effect.
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo
*
* "pgstat_report_fixed" needs to be set to trigger the flush of pending
* stats.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_static_cb) (bool nowait);
+ bool (*flush_static_cb) (bool nowait, bool anytime_only);
/*
* For fixed-numbered statistics: Reset All.
@@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
@@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
-extern bool pgstat_flush_backend(bool nowait, bits32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
+extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only);
+extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_io.c
*/
-extern void pgstat_flush_io(bool nowait);
+extern void pgstat_flush_io(bool nowait, bool anytime_only);
-extern bool pgstat_io_flush_cb(bool nowait);
+extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
* Functions in pgstat_slru.c
*/
-extern bool pgstat_slru_flush_cb(bool nowait);
+extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void);
*/
extern void pgstat_wal_init_backend_cb(void);
-extern bool pgstat_wal_flush_cb(bool nowait);
+extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
@@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 64a8fe63cce..bc0b5d6e0eb 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait);
+ bool nowait, bool anytime_only);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -150,7 +150,7 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1dbc4b96f51 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2271,6 +2271,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v7-0002-Add-anytime-flush-tests-for-custom-stats.patch (9.0K, ../../aZLVDJLTiFlxsWZi@ip-10-97-1-34.eu-west-3.compute.internal/3-v7-0002-Add-anytime-flush-tests-for-custom-stats.patch)
download | inline diff:
From 9a2b8b55eca91595b37912da8e102ff1aa109e62 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 5 Feb 2026 05:54:34 +0000
Subject: [PATCH v7 2/5] Add anytime flush tests for custom stats
---
.../test_custom_stats/t/001_custom_stats.pl | 41 +++++++++++++
.../test_custom_fixed_stats--1.0.sql | 5 ++
.../test_custom_fixed_stats.c | 57 +++++++++++++++++++
.../test_custom_var_stats--1.0.sql | 5 ++
.../test_custom_stats/test_custom_var_stats.c | 27 +++++++++
5 files changed, 135 insertions(+)
33.8% src/test/modules/test_custom_stats/t/
66.1% src/test/modules/test_custom_stats/
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 9e6a7a38577..7be1b281776 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -156,5 +156,46 @@ $result = $node->safe_psql('postgres',
);
is($result, "0", "report of fixed-sized after manual reset");
+# Test FLUSH_ANYTIME mechanism with custom fixed stats
+# This verifies that custom stats can be flushed during a transaction
+
+# Reset stats first
+$node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+my $anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select 'anytime:'||numcalls from test_custom_stats_fixed_report();
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^anytime:2/m,
+ "anytime fixed stats flushed during transaction");
+
+# Test FLUSH_ANYTIME mechanism with custom variable stats
+# This verifies that custom stats can be flushed during a transaction
+
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+$anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_var_anytime_update('entry2');
+ select test_custom_stats_var_anytime_update('entry2');
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select * from test_custom_stats_var_report('entry2');
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^entry2|2|/m,
+ "anytime var stats flushed during transaction");
+
# Test completed successfully
done_testing();
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
index 69a93b5241f..da3a798f289 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
@@ -18,3 +18,8 @@ CREATE FUNCTION test_custom_stats_fixed_reset()
RETURNS void
AS 'MODULE_PATHNAME', 'test_custom_stats_fixed_reset'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_fixed_anytime_update()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
index 908bd18a7c7..30b0fbcbdc7 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
@@ -18,6 +18,7 @@
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_fixed_stats",
@@ -43,11 +44,13 @@ typedef struct PgStatShared_CustomFixedEntry
static void test_custom_stats_fixed_init_shmem_cb(void *stats);
static void test_custom_stats_fixed_reset_all_cb(TimestampTz ts);
static void test_custom_stats_fixed_snapshot_cb(void);
+static bool test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only);
static const PgStat_KindInfo custom_stats = {
.name = "test_custom_fixed_stats",
.fixed_amount = true, /* exactly one entry */
.write_to_file = true, /* persist to stats file */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.shared_size = sizeof(PgStat_StatCustomFixedEntry),
.shared_data_off = offsetof(PgStatShared_CustomFixedEntry, stats),
@@ -56,8 +59,12 @@ static const PgStat_KindInfo custom_stats = {
.init_shmem_cb = test_custom_stats_fixed_init_shmem_cb,
.reset_all_cb = test_custom_stats_fixed_reset_all_cb,
.snapshot_cb = test_custom_stats_fixed_snapshot_cb,
+ .flush_static_cb = test_custom_stats_fixed_flush_cb,
};
+/* Pending statistics */
+static PgStat_StatCustomFixedEntry PendingCustomStats = {0};
+
/*
* Kind ID for test_custom_fixed_stats.
*/
@@ -141,6 +148,38 @@ test_custom_stats_fixed_snapshot_cb(void)
#undef FIXED_COMP
}
+/*
+ * test_custom_stats_fixed_flush_cb
+ * Flush pending stats to shared memory
+ */
+static bool
+test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only)
+{
+ PgStatShared_CustomFixedEntry *stats_shmem;
+
+ /* Nothing to flush if no calls were made */
+ if (PendingCustomStats.numcalls == 0)
+ return false;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_TEST_CUSTOM_FIXED_STATS);
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numcalls += PendingCustomStats.numcalls;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ LWLockRelease(&stats_shmem->lock);
+
+ /* Reset pending stats */
+ PendingCustomStats.numcalls = 0;
+
+ return false; /* successfully flushed */
+}
+
/*--------------------------------------------------------------------------
* SQL-callable functions
*--------------------------------------------------------------------------
@@ -222,3 +261,21 @@ test_custom_stats_fixed_report(PG_FUNCTION_ARGS)
/* Return as tuple */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/*
+ * test_custom_stats_fixed_anytime_update
+ * Increment call counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_fixed_anytime_update);
+Datum
+test_custom_stats_fixed_anytime_update(PG_FUNCTION_ARGS)
+{
+ /* Accumulate in pending stats */
+ PendingCustomStats.numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+ pgstat_report_fixed = true;
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
index 5ed8cfc2dcf..ed66d38981e 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
@@ -24,3 +24,8 @@ CREATE FUNCTION test_custom_stats_var_report(INOUT name TEXT,
RETURNS SETOF record
AS 'MODULE_PATHNAME', 'test_custom_stats_var_report'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_var_anytime_update(IN name TEXT)
+RETURNS void
+AS 'MODULE_PATHNAME', 'test_custom_stats_var_anytime_update'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index bc0b5d6e0eb..207e841911b 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -17,6 +17,7 @@
#include "storage/dsm_registry.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_var_stats",
@@ -107,6 +108,7 @@ static const PgStat_KindInfo custom_stats = {
.name = "test_custom_var_stats",
.fixed_amount = false, /* variable number of entries */
.write_to_file = true, /* persist across restarts */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.track_entry_count = true, /* count active entries */
.accessed_across_databases = true, /* global statistics */
.shared_size = sizeof(PgStatShared_CustomVarEntry),
@@ -689,3 +691,28 @@ test_custom_stats_var_report(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * test_custom_stats_var_anytime_update
+ * Increment custom statistic counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_var_anytime_update);
+Datum
+test_custom_stats_var_anytime_update(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_EntryRef *entry_ref;
+ PgStat_StatCustomVarEntry *pending_entry;
+
+ /* Get pending entry in local memory */
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, InvalidOid,
+ PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name), NULL);
+
+ pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
+ pending_entry->numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+
+ PG_RETURN_VOID();
+}
--
2.34.1
[text/x-diff] v7-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch (8.9K, ../../aZLVDJLTiFlxsWZi@ip-10-97-1-34.eu-west-3.compute.internal/4-v7-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch)
download | inline diff:
From 347b711e8daaac17e68da9240ddcb9589ea0deaf Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v7 3/5] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 32 +++++++++++++++++++
src/backend/utils/activity/pgstat.c | 16 ++++++++++
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/pgstat.h | 6 ++--
src/include/utils/guc_hooks.h | 1 +
.../test_custom_stats/t/001_custom_stats.pl | 6 ++--
7 files changed, 66 insertions(+), 6 deletions(-)
51.8% doc/src/sgml/
13.3% src/backend/utils/activity/
13.6% src/backend/utils/misc/
11.3% src/include/
9.8% src/test/modules/test_custom_stats/t/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc2690ce07..383bbe3a132 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8923,6 +8923,38 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which certain statistics, which can be updated while a
+ transaction is in progress, are made visible. These include WAL activity
+ and I/O operations.
+ Such statistics are refreshed at the specified interval and can be observed
+ during active transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>
+ and
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is generally
+ the smallest practical value for long-running transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect statistics that are only reported at
+ transaction end, such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics are always
+ flushed at the end of a transaction.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index a4ff64dc5ce..dd85a27c52f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -123,6 +123,8 @@
* ----------
*/
+/* minimum interval non-forced stats flushes.*/
+#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -203,6 +205,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -2164,6 +2167,19 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ {
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+ }
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 271c033952e..d2734caafea 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2801,6 +2801,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f938cc65a3a..8bd37a25b38 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -688,6 +688,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b340a680614..ef856dbf55b 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,9 +35,6 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
-/* Minimum interval non-forced stats flushes */
-#define PGSTAT_MIN_INTERVAL 1000
-
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -548,7 +545,7 @@ extern void pgstat_force_next_flush(void);
#define pgstat_schedule_anytime_update() \
do { \
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval); \
} while (0)
extern void pgstat_reset_counters(void);
@@ -828,6 +825,7 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 9c90670d9b8..9b5d2a90387 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 7be1b281776..22e2a75dcb9 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -164,10 +164,11 @@ $node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
my $anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
-- Accumulate stats
select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select 'anytime:'||numcalls from test_custom_stats_fixed_report();
@@ -183,11 +184,12 @@ like($result, qr/^anytime:2/m,
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
$anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
-- Accumulate stats
select test_custom_stats_var_anytime_update('entry2');
select test_custom_stats_var_anytime_update('entry2');
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select * from test_custom_stats_var_report('entry2');
--
2.34.1
[text/x-diff] v7-0004-Remove-useless-calls-to-flush-some-stats.patch (7.6K, ../../aZLVDJLTiFlxsWZi@ip-10-97-1-34.eu-west-3.compute.internal/5-v7-0004-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From 96672df86139c22efbf8d8d320b1e70c8f9fba5a Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v7 4/5] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v7-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch (34.2K, ../../aZLVDJLTiFlxsWZi@ip-10-97-1-34.eu-west-3.compute.internal/6-v7-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch)
download | inline diff:
From 2914134d2386d469bd48e999a6e2b17e6f7efea7 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v7 5/5] Change RELATION and DATABASE stats to anytime flush
This commit allows mixing fields with different transaction behavior within
the same RELATION or DATABASE statistics kind: some fields are transactional
(e.g., tuple inserts/updates/deletes) while others are non-transactional
(e.g., sequential scans, blocks read).
It modifies the relation flush callback to handle the anytime_only parameter
introduced in commit <nnnn>.
Implementation details:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Change DATABASE from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Add a is_partial parameter to flush_pending_cb() to be able to distinguish
partial flushes in pgstat_flush_pending_entries()
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats. When set to true, it clears flushed fields from pending stats to
prevent double-counting at transaction boundary
DATABASE stats inherit the anytime flush behavior so that relation-derived
stats (tuples_returned, tuples_fetched, blocks_fetched, blocks_hit) are
visible while transactions are in progress.
Tests are added to verify the anytime flush behavior for mixed fields.
---
doc/src/sgml/monitoring.sgml | 37 ++++++-
src/backend/utils/activity/pgstat.c | 15 +--
src/backend/utils/activity/pgstat_database.c | 6 +-
src/backend/utils/activity/pgstat_function.c | 6 +-
src/backend/utils/activity/pgstat_relation.c | 92 ++++++++++++----
.../utils/activity/pgstat_subscription.c | 6 +-
src/include/pgstat.h | 27 ++++-
src/include/utils/pgstat_internal.h | 16 ++-
src/test/isolation/expected/stats.out | 102 ++++++++++++++++++
src/test/isolation/expected/stats_1.out | 102 ++++++++++++++++++
src/test/isolation/specs/stats.spec | 27 ++++-
.../test_custom_stats/test_custom_var_stats.c | 9 +-
12 files changed, 404 insertions(+), 41 deletions(-)
11.7% doc/src/sgml/
26.8% src/backend/utils/activity/
4.2% src/include/utils/
5.4% src/include/
45.1% src/test/isolation/expected/
4.7% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..f2321b631b0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -3956,8 +3969,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_seq_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last sequential scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last sequential scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -3984,8 +3997,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_idx_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last index scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last index scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index dd85a27c52f..a20a87709c6 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -291,7 +291,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -309,7 +309,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -1347,7 +1347,8 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries. For entries
+ * that support it, the callback may flush only non-transactional fields.
* This is safe to call inside transactions.
*
* If anytime_only is false, flushes all entries.
@@ -1378,6 +1379,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1398,7 +1400,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
}
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait,
+ anytime_only, &is_partial_flush);
Assert(did_flush || nowait);
@@ -1408,8 +1411,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 8e86df60461..59dd0790fd7 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,8 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
@@ -443,6 +444,9 @@ pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
pendingent = (PgStat_StatDBEntry *) entry_ref->pending;
sharedent = (PgStatShared_Database *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index 5ba4958382f..44193c93fc7 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,7 +190,8 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
@@ -200,6 +201,9 @@ pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ae2952cae89..62363dacfe1 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -47,7 +47,19 @@ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_lev
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
+static void flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry,
+ PgStat_TableCounts *counts, bool anytime_only);
+/*
+ * Update database statistics with non-transactional stats.
+ */
+#define UPDATE_DATABASE_ANYTIME_STATS(dbentry, counts) \
+ do { \
+ (dbentry)->tuples_returned += (counts)->tuples_returned; \
+ (dbentry)->tuples_fetched += (counts)->tuples_fetched; \
+ (dbentry)->blocks_fetched += (counts)->blocks_fetched; \
+ (dbentry)->blocks_hit += (counts)->blocks_hit; \
+ } while (0)
/*
* Copy stats between relations. This is used for things like REINDEX
@@ -789,6 +801,29 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Helper function to flush non-transactional statistics.
+ */
+static void
+flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry, PgStat_TableCounts *counts,
+ bool anytime_only)
+{
+ TimestampTz t;
+
+ tabentry->numscans += counts->numscans;
+ if (counts->numscans)
+ {
+ t = anytime_only ? GetCurrentTimestamp() : GetCurrentTransactionStopTimestamp();
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+
+ tabentry->tuples_returned += counts->tuples_returned;
+ tabentry->tuples_fetched += counts->tuples_fetched;
+ tabentry->blocks_fetched += counts->blocks_fetched;
+ tabentry->blocks_hit += counts->blocks_hit;
+}
+
/*
* Flush out pending stats for the entry
*
@@ -797,9 +832,17 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
*
* Some of the stats are copied to the corresponding pending database stats
* entry when successfully flushing.
+ *
+ * If anytime_only is true, only non-transactional fields are flushed
+ * (numscans, tuples_returned, tuples_fetched, blocks_fetched, blocks_hit).
+ * Transactional fields remain pending until transaction boundary.
+ *
+ * Some of the stats are copied to the corresponding pending database stats
+ * entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -807,12 +850,13 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
- Assert(!anytime_only);
-
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+ /* this is a partial flush if in anytime only mode */
+ *is_partial = anytime_only;
+
/*
* Ignore entries that didn't accumulate any actual counts, such as
* indexes that were opened by the planner but not used.
@@ -824,19 +868,36 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
- /* add the values to the shared entry. */
tabentry = &shtabstats->stats;
- tabentry->numscans += lstats->counts.numscans;
- if (lstats->counts.numscans)
+ if (anytime_only)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
- if (t > tabentry->lastscan)
- tabentry->lastscan = t;
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, true);
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
+ /*
+ * Clear the flushed fields from pending stats to prevent
+ * double-counting when we flush all fields at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
}
- tabentry->tuples_returned += lstats->counts.tuples_returned;
- tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, false);
+
tabentry->tuples_inserted += lstats->counts.tuples_inserted;
tabentry->tuples_updated += lstats->counts.tuples_updated;
tabentry->tuples_deleted += lstats->counts.tuples_deleted;
@@ -866,9 +927,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
*/
tabentry->ins_since_vacuum += lstats->counts.tuples_inserted;
- tabentry->blocks_fetched += lstats->counts.blocks_fetched;
- tabentry->blocks_hit += lstats->counts.blocks_hit;
-
/* Clamp live_tuples in case of negative delta_live_tuples */
tabentry->live_tuples = Max(tabentry->live_tuples, 0);
/* Likewise for dead_tuples */
@@ -878,13 +936,11 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
/* The entry was successfully flushed, add the same to database stats */
dbentry = pgstat_prep_database_pending(dboid);
- dbentry->tuples_returned += lstats->counts.tuples_returned;
- dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
dbentry->tuples_inserted += lstats->counts.tuples_inserted;
dbentry->tuples_updated += lstats->counts.tuples_updated;
dbentry->tuples_deleted += lstats->counts.tuples_deleted;
- dbentry->blocks_fetched += lstats->counts.blocks_fetched;
- dbentry->blocks_hit += lstats->counts.blocks_hit;
return true;
}
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index c4614817966..43fec86c635 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -116,7 +116,8 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
@@ -126,6 +127,9 @@ pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anyti
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ef856dbf55b..06639198f28 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -21,6 +21,7 @@
#include "utils/backend_status.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/pgstat_kind.h"
#include "utils/relcache.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h" /* for backward compatibility */ /* IWYU pragma: export */
@@ -537,10 +538,11 @@ extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
/*
- * Schedule the next anytime stats update timeout.
+ * Schedule the next anytime stats update timeout and mark that we have
+ * mixed anytime stats pending.
*
* This should be called whenever accumulating statistics that support
- * FLUSH_ANYTIME flushing mode.
+ * FLUSH_ANYTIME or FLUSH_MIXED flushing modes.
*/
#define pgstat_schedule_anytime_update() \
do { \
@@ -703,37 +705,58 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned += (n); \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_hit++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 607f4255268..1a2114aad8a 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -322,8 +322,10 @@ typedef struct PgStat_KindInfo
* that cannot use PgStat_EntryRef->pending.
*
* The anytime_only parameter indicates whether this is an anytime flush.
+ * The is_partial parameter indicates whether this is a partial flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait,
+ bool anytime_only, bool *is_partial);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -757,7 +759,8 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -765,7 +768,8 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -790,7 +794,8 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -858,7 +863,8 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..11e3e57806d 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..aef582e7582 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..47414eb6009 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -156,6 +162,8 @@ step s2_func_stats {
}
# Relation stats steps
+step s2_track_counts_on { SET track_counts = on; }
+step s2_track_counts_off { SET track_counts = off; }
step s2_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; }
step s2_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';}
@@ -435,6 +443,23 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_off
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_on
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 207e841911b..ffcda7b6c7a 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -84,7 +84,8 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait, bool anytime_only);
+ bool nowait, bool anytime_only,
+ bool *is_partial);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -152,7 +153,8 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
@@ -160,6 +162,9 @@ test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
shared_entry = (PgStatShared_CustomVarEntry *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-16 22:42 ` Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-02-16 22:42 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Thanks!
> > 2/
> > I don't see we have tests for other timeout based GUCs, but it would nice
> > to ensure that this woks correctly. Maybe as a custom_stats test where we
> > SET stats_flush_interval inside the transaction and make sure the stats flush
> > only after the new timeout has passed. Maybe?
>
> Not sure I follow, that's what 0002 is doing.
I was referring to "SET stats_flush_interval", which you are doing in 0005,
so disregard this comment. I wrote this comment when I read 0003 only.
> > Will this be tue at all time? Let's imagine a Kind that flushes all the fields
> > ANYTIME, would we not want to delete the pending entry?
> >
> > + /* if successfull non-partial flush, remove entry */
> > + if (did_flush && !is_partial_flush)
> > pgstat_delete_pending_entry(entry_ref);
> >
>
> Right, and that's why the MIXED flush mode was useful, i.e to be able to distinguish
> here. So, in the attached, instead of re-introducing the MIXED flush mode, I added
> a "is_partial" bool paramater to the flush_pending_cb().
>
Yeah, MIXED flush mode to deal with this is still not the right idea, IMO.
I do agree with what you did here by passing "is_partial" to the callbacks.
Now it will be up to the callbacks to determine if they performed a partial
or a complete flush ( even in the case of any anytime flush, if they so
happened to flush stats for all the fields ).
I do not have any further comments on this patchset.
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-17 07:04 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-17 07:04 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Mon, Feb 16, 2026 at 04:42:50PM -0600, Sami Imseih wrote:
> I do agree with what you did here by passing "is_partial" to the callbacks.
> Now it will be up to the callbacks to determine if they performed a partial
> or a complete flush ( even in the case of any anytime flush, if they so
> happened to flush stats for all the fields ).
Right.
> I do not have any further comments on this patchset.
Thanks for the review!
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-17 19:18 ` Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-02-17 19:18 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
>
> > I do not have any further comments on this patchset.
>
> Thanks for the review!
I flipped this CF entry to Ready-for-committer
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-18 05:40 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-18 11:37 ` Re: Flush some statistics within running transactions Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-18 05:40 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Tue, Feb 17, 2026 at 01:18:35PM -0600, Sami Imseih wrote:
> >
> > > I do not have any further comments on this patchset.
> >
> > Thanks for the review!
>
> I flipped this CF entry to Ready-for-committer
Thanks!
PFA a mandatory rebase (nothing that needs review) due to a92b809f9da1.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v8-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (41.9K, ../../aZVQ3nT9B6ti8whJ@ip-10-97-1-34.eu-west-3.compute.internal/2-v8-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From fc1f5423f49d14850b9dde764caade5d633df8bd Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v8 1/5] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces
spikes when flushed.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducin
a new callback).
- Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush,
relying on PGSTAT_MIN_INTERVAL
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 6 +
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/replication/walsender.c | 8 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 112 ++++++++++++++----
src/backend/utils/activity/pgstat_backend.c | 13 +-
src/backend/utils/activity/pgstat_bgwriter.c | 2 +-
.../utils/activity/pgstat_checkpointer.c | 2 +-
src/backend/utils/activity/pgstat_database.c | 2 +-
src/backend/utils/activity/pgstat_function.c | 4 +-
src/backend/utils/activity/pgstat_io.c | 10 +-
src/backend/utils/activity/pgstat_relation.c | 12 +-
src/backend/utils/activity/pgstat_slru.c | 6 +-
.../utils/activity/pgstat_subscription.c | 4 +-
src/backend/utils/activity/pgstat_wal.c | 10 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 16 +++
src/include/utils/pgstat_internal.h | 52 ++++++--
src/include/utils/timeout.h | 1 +
.../test_custom_stats/test_custom_var_stats.c | 4 +-
src/tools/pgindent/typedefs.list | 1 +
28 files changed, 264 insertions(+), 66 deletions(-)
10.8% src/backend/postmaster/
6.0% src/backend/replication/
50.7% src/backend/utils/activity/
6.0% src/backend/
19.3% src/include/utils/
5.6% src/include/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..d01b11c7470 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0956bd39a85..059c601c3b8 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..e11c4b099c8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..4954fe425b7 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 742137edad6..f1bae9d23d6 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN); /* no query to cancel */
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 7c0e2809c17..bcf59227a00 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN); /* no query to cancel */
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 10e64a7d1f4..11b7c114d3b 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2cde8ebc729..a7214d0dc6f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
@@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 21de158adbb..2089de782d5 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3564,6 +3564,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..a4ff64dc5ce 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -112,6 +112,7 @@
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +123,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +784,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
@@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
@@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait, anytime_only);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
+
+/*
+ * Timeout handler for flushing anytime stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index f2f8d3ff75f..b09316d3ab3 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
* if some statistics could not be flushed due to lock contention.
*/
bool
-pgstat_flush_backend(bool nowait, bits32 flags)
+pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only)
{
PgStat_EntryRef *entry_ref;
bool has_pending_data = false;
@@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
* If some stats could not be flushed due to lock contention, return true.
*/
bool
-pgstat_backend_flush_cb(bool nowait)
+pgstat_backend_flush_cb(bool nowait, bool anytime_only)
{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
+ return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only);
}
/*
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index ed2fd801189..1c5f0c3ec40 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -61,7 +61,7 @@ pgstat_report_bgwriter(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 1f70194b7a7..2d89a082464 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -68,7 +68,7 @@ pgstat_report_checkpointer(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 933dcb5cae5..8e86df60461 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index e6b84283c6c..5ba4958382f 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
+ Assert(!anytime_only);
+
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..7cd32900236 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void)
* Simpler wrapper of pgstat_io_flush_cb()
*/
void
-pgstat_flush_io(bool nowait)
+pgstat_flush_io(bool nowait, bool anytime_only)
{
- (void) pgstat_io_flush_cb(nowait);
+ (void) pgstat_io_flush_cb(nowait, anytime_only);
}
/*
@@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait)
* acquired. Otherwise, return false.
*/
bool
-pgstat_io_flush_cb(bool nowait)
+pgstat_io_flush_cb(bool nowait, bool anytime_only)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..04d21483d93 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* is done -- which will likely vacuum many relations -- or until the
* VACUUM command has processed all tables and committed.
*/
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel,
pgstat_unlock_entry(entry_ref);
/* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
* entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
+ Assert(!anytime_only);
+
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..bf8a4d58673 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name)
* acquired. Otherwise return false.
*/
bool
-pgstat_slru_flush_cb(bool nowait)
+pgstat_slru_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
int i;
@@ -223,6 +224,9 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 500b1899188..c4614817966 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
+ Assert(!anytime_only);
+
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..2c2f3f10e10 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -51,12 +51,12 @@ pgstat_report_wal(bool force)
nowait = !force;
/* flush wal stats */
- (void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
+ (void) pgstat_wal_flush_cb(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
/* flush IO stats */
- pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -88,7 +88,7 @@ pgstat_wal_have_pending(void)
* acquired. Otherwise return false.
*/
bool
-pgstat_wal_flush_cb(bool nowait)
+pgstat_wal_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
WalUsage wal_usage_diff = {0};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..eeeac1bf39a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..84e698da214 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..b340a680614 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,8 +536,21 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
+/*
+ * Schedule the next anytime stats update timeout.
+ *
+ * This should be called whenever accumulating statistics that support
+ * FLUSH_ANYTIME flushing mode.
+ */
+#define pgstat_schedule_anytime_update() \
+ do { \
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ } while (0)
+
extern void pgstat_reset_counters(void);
extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_reset_of_kind(PgStat_Kind kind);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..607f4255268 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * The mode of when to flush stats. See PgStat_FlushMode for more details.
+ *
+ * This member only has meaning for statistics kinds that accumulate
+ * pending stats and use flush callbacks. For kinds that write directly to
+ * shared memory (e.g., archiver, bgwriter, checkpointer), this member has
+ * no effect.
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo
*
* "pgstat_report_fixed" needs to be set to trigger the flush of pending
* stats.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_static_cb) (bool nowait);
+ bool (*flush_static_cb) (bool nowait, bool anytime_only);
/*
* For fixed-numbered statistics: Reset All.
@@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
@@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
-extern bool pgstat_flush_backend(bool nowait, bits32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
+extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only);
+extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_io.c
*/
-extern void pgstat_flush_io(bool nowait);
+extern void pgstat_flush_io(bool nowait, bool anytime_only);
-extern bool pgstat_io_flush_cb(bool nowait);
+extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
* Functions in pgstat_slru.c
*/
-extern bool pgstat_slru_flush_cb(bool nowait);
+extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void);
*/
extern void pgstat_wal_init_backend_cb(void);
-extern bool pgstat_wal_flush_cb(bool nowait);
+extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
@@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 64a8fe63cce..bc0b5d6e0eb 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait);
+ bool nowait, bool anytime_only);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -150,7 +150,7 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1dbc4b96f51 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2271,6 +2271,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v8-0002-Add-anytime-flush-tests-for-custom-stats.patch (9.0K, ../../aZVQ3nT9B6ti8whJ@ip-10-97-1-34.eu-west-3.compute.internal/3-v8-0002-Add-anytime-flush-tests-for-custom-stats.patch)
download | inline diff:
From 3f426eb4b56382aacfb7e3ae8377d7c8fae3db91 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 5 Feb 2026 05:54:34 +0000
Subject: [PATCH v8 2/5] Add anytime flush tests for custom stats
---
.../test_custom_stats/t/001_custom_stats.pl | 41 +++++++++++++
.../test_custom_fixed_stats--1.0.sql | 5 ++
.../test_custom_fixed_stats.c | 57 +++++++++++++++++++
.../test_custom_var_stats--1.0.sql | 5 ++
.../test_custom_stats/test_custom_var_stats.c | 27 +++++++++
5 files changed, 135 insertions(+)
33.8% src/test/modules/test_custom_stats/t/
66.1% src/test/modules/test_custom_stats/
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 9e6a7a38577..7be1b281776 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -156,5 +156,46 @@ $result = $node->safe_psql('postgres',
);
is($result, "0", "report of fixed-sized after manual reset");
+# Test FLUSH_ANYTIME mechanism with custom fixed stats
+# This verifies that custom stats can be flushed during a transaction
+
+# Reset stats first
+$node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+my $anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select 'anytime:'||numcalls from test_custom_stats_fixed_report();
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^anytime:2/m,
+ "anytime fixed stats flushed during transaction");
+
+# Test FLUSH_ANYTIME mechanism with custom variable stats
+# This verifies that custom stats can be flushed during a transaction
+
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+$anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_var_anytime_update('entry2');
+ select test_custom_stats_var_anytime_update('entry2');
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select * from test_custom_stats_var_report('entry2');
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^entry2|2|/m,
+ "anytime var stats flushed during transaction");
+
# Test completed successfully
done_testing();
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
index 69a93b5241f..da3a798f289 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
@@ -18,3 +18,8 @@ CREATE FUNCTION test_custom_stats_fixed_reset()
RETURNS void
AS 'MODULE_PATHNAME', 'test_custom_stats_fixed_reset'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_fixed_anytime_update()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
index 908bd18a7c7..30b0fbcbdc7 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
@@ -18,6 +18,7 @@
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_fixed_stats",
@@ -43,11 +44,13 @@ typedef struct PgStatShared_CustomFixedEntry
static void test_custom_stats_fixed_init_shmem_cb(void *stats);
static void test_custom_stats_fixed_reset_all_cb(TimestampTz ts);
static void test_custom_stats_fixed_snapshot_cb(void);
+static bool test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only);
static const PgStat_KindInfo custom_stats = {
.name = "test_custom_fixed_stats",
.fixed_amount = true, /* exactly one entry */
.write_to_file = true, /* persist to stats file */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.shared_size = sizeof(PgStat_StatCustomFixedEntry),
.shared_data_off = offsetof(PgStatShared_CustomFixedEntry, stats),
@@ -56,8 +59,12 @@ static const PgStat_KindInfo custom_stats = {
.init_shmem_cb = test_custom_stats_fixed_init_shmem_cb,
.reset_all_cb = test_custom_stats_fixed_reset_all_cb,
.snapshot_cb = test_custom_stats_fixed_snapshot_cb,
+ .flush_static_cb = test_custom_stats_fixed_flush_cb,
};
+/* Pending statistics */
+static PgStat_StatCustomFixedEntry PendingCustomStats = {0};
+
/*
* Kind ID for test_custom_fixed_stats.
*/
@@ -141,6 +148,38 @@ test_custom_stats_fixed_snapshot_cb(void)
#undef FIXED_COMP
}
+/*
+ * test_custom_stats_fixed_flush_cb
+ * Flush pending stats to shared memory
+ */
+static bool
+test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only)
+{
+ PgStatShared_CustomFixedEntry *stats_shmem;
+
+ /* Nothing to flush if no calls were made */
+ if (PendingCustomStats.numcalls == 0)
+ return false;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_TEST_CUSTOM_FIXED_STATS);
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numcalls += PendingCustomStats.numcalls;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ LWLockRelease(&stats_shmem->lock);
+
+ /* Reset pending stats */
+ PendingCustomStats.numcalls = 0;
+
+ return false; /* successfully flushed */
+}
+
/*--------------------------------------------------------------------------
* SQL-callable functions
*--------------------------------------------------------------------------
@@ -222,3 +261,21 @@ test_custom_stats_fixed_report(PG_FUNCTION_ARGS)
/* Return as tuple */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/*
+ * test_custom_stats_fixed_anytime_update
+ * Increment call counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_fixed_anytime_update);
+Datum
+test_custom_stats_fixed_anytime_update(PG_FUNCTION_ARGS)
+{
+ /* Accumulate in pending stats */
+ PendingCustomStats.numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+ pgstat_report_fixed = true;
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
index 5ed8cfc2dcf..ed66d38981e 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
@@ -24,3 +24,8 @@ CREATE FUNCTION test_custom_stats_var_report(INOUT name TEXT,
RETURNS SETOF record
AS 'MODULE_PATHNAME', 'test_custom_stats_var_report'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_var_anytime_update(IN name TEXT)
+RETURNS void
+AS 'MODULE_PATHNAME', 'test_custom_stats_var_anytime_update'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index bc0b5d6e0eb..207e841911b 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -17,6 +17,7 @@
#include "storage/dsm_registry.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_var_stats",
@@ -107,6 +108,7 @@ static const PgStat_KindInfo custom_stats = {
.name = "test_custom_var_stats",
.fixed_amount = false, /* variable number of entries */
.write_to_file = true, /* persist across restarts */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.track_entry_count = true, /* count active entries */
.accessed_across_databases = true, /* global statistics */
.shared_size = sizeof(PgStatShared_CustomVarEntry),
@@ -689,3 +691,28 @@ test_custom_stats_var_report(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * test_custom_stats_var_anytime_update
+ * Increment custom statistic counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_var_anytime_update);
+Datum
+test_custom_stats_var_anytime_update(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_EntryRef *entry_ref;
+ PgStat_StatCustomVarEntry *pending_entry;
+
+ /* Get pending entry in local memory */
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, InvalidOid,
+ PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name), NULL);
+
+ pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
+ pending_entry->numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+
+ PG_RETURN_VOID();
+}
--
2.34.1
[text/x-diff] v8-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch (8.9K, ../../aZVQ3nT9B6ti8whJ@ip-10-97-1-34.eu-west-3.compute.internal/4-v8-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch)
download | inline diff:
From 1be773551abc042e941b538dc472294869718b3a Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v8 3/5] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 32 +++++++++++++++++++
src/backend/utils/activity/pgstat.c | 16 ++++++++++
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/pgstat.h | 6 ++--
src/include/utils/guc_hooks.h | 1 +
.../test_custom_stats/t/001_custom_stats.pl | 6 ++--
7 files changed, 66 insertions(+), 6 deletions(-)
51.8% doc/src/sgml/
13.3% src/backend/utils/activity/
13.6% src/backend/utils/misc/
11.3% src/include/
9.8% src/test/modules/test_custom_stats/t/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index faf0bdb62aa..03875b490b7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8932,6 +8932,38 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which certain statistics, which can be updated while a
+ transaction is in progress, are made visible. These include WAL activity
+ and I/O operations.
+ Such statistics are refreshed at the specified interval and can be observed
+ during active transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>
+ and
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is generally
+ the smallest practical value for long-running transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect statistics that are only reported at
+ transaction end, such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics are always
+ flushed at the end of a transaction.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index a4ff64dc5ce..dd85a27c52f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -123,6 +123,8 @@
* ----------
*/
+/* minimum interval non-forced stats flushes.*/
+#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -203,6 +205,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -2164,6 +2167,19 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ {
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+ }
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 271c033952e..d2734caafea 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2801,6 +2801,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f938cc65a3a..8bd37a25b38 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -688,6 +688,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b340a680614..ef856dbf55b 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,9 +35,6 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
-/* Minimum interval non-forced stats flushes */
-#define PGSTAT_MIN_INTERVAL 1000
-
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -548,7 +545,7 @@ extern void pgstat_force_next_flush(void);
#define pgstat_schedule_anytime_update() \
do { \
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval); \
} while (0)
extern void pgstat_reset_counters(void);
@@ -828,6 +825,7 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 9c90670d9b8..9b5d2a90387 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 7be1b281776..22e2a75dcb9 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -164,10 +164,11 @@ $node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
my $anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
-- Accumulate stats
select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select 'anytime:'||numcalls from test_custom_stats_fixed_report();
@@ -183,11 +184,12 @@ like($result, qr/^anytime:2/m,
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
$anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
-- Accumulate stats
select test_custom_stats_var_anytime_update('entry2');
select test_custom_stats_var_anytime_update('entry2');
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select * from test_custom_stats_var_report('entry2');
--
2.34.1
[text/x-diff] v8-0004-Remove-useless-calls-to-flush-some-stats.patch (7.6K, ../../aZVQ3nT9B6ti8whJ@ip-10-97-1-34.eu-west-3.compute.internal/5-v8-0004-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From 3938754d5ab37da226ad86278ad1a104f82e80ed Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v8 4/5] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v8-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch (34.2K, ../../aZVQ3nT9B6ti8whJ@ip-10-97-1-34.eu-west-3.compute.internal/6-v8-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch)
download | inline diff:
From b3d0b50b0c3a258adf61db81464a2910af0f68c4 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v8 5/5] Change RELATION and DATABASE stats to anytime flush
This commit allows mixing fields with different transaction behavior within
the same RELATION or DATABASE statistics kind: some fields are transactional
(e.g., tuple inserts/updates/deletes) while others are non-transactional
(e.g., sequential scans, blocks read).
It modifies the relation flush callback to handle the anytime_only parameter
introduced in commit <nnnn>.
Implementation details:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Change DATABASE from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Add a is_partial parameter to flush_pending_cb() to be able to distinguish
partial flushes in pgstat_flush_pending_entries()
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats. When set to true, it clears flushed fields from pending stats to
prevent double-counting at transaction boundary
DATABASE stats inherit the anytime flush behavior so that relation-derived
stats (tuples_returned, tuples_fetched, blocks_fetched, blocks_hit) are
visible while transactions are in progress.
Tests are added to verify the anytime flush behavior for mixed fields.
---
doc/src/sgml/monitoring.sgml | 37 ++++++-
src/backend/utils/activity/pgstat.c | 15 +--
src/backend/utils/activity/pgstat_database.c | 6 +-
src/backend/utils/activity/pgstat_function.c | 6 +-
src/backend/utils/activity/pgstat_relation.c | 92 ++++++++++++----
.../utils/activity/pgstat_subscription.c | 6 +-
src/include/pgstat.h | 27 ++++-
src/include/utils/pgstat_internal.h | 16 ++-
src/test/isolation/expected/stats.out | 102 ++++++++++++++++++
src/test/isolation/expected/stats_1.out | 102 ++++++++++++++++++
src/test/isolation/specs/stats.spec | 27 ++++-
.../test_custom_stats/test_custom_var_stats.c | 9 +-
12 files changed, 404 insertions(+), 41 deletions(-)
11.7% doc/src/sgml/
26.8% src/backend/utils/activity/
4.2% src/include/utils/
5.4% src/include/
45.1% src/test/isolation/expected/
4.7% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..f2321b631b0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -3956,8 +3969,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_seq_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last sequential scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last sequential scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -3984,8 +3997,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_idx_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last index scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last index scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index dd85a27c52f..a20a87709c6 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -291,7 +291,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -309,7 +309,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -1347,7 +1347,8 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries. For entries
+ * that support it, the callback may flush only non-transactional fields.
* This is safe to call inside transactions.
*
* If anytime_only is false, flushes all entries.
@@ -1378,6 +1379,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1398,7 +1400,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
}
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait,
+ anytime_only, &is_partial_flush);
Assert(did_flush || nowait);
@@ -1408,8 +1411,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 8e86df60461..59dd0790fd7 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,8 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
@@ -443,6 +444,9 @@ pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
pendingent = (PgStat_StatDBEntry *) entry_ref->pending;
sharedent = (PgStatShared_Database *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index 5ba4958382f..44193c93fc7 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,7 +190,8 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
@@ -200,6 +201,9 @@ pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ae2952cae89..62363dacfe1 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -47,7 +47,19 @@ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_lev
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
+static void flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry,
+ PgStat_TableCounts *counts, bool anytime_only);
+/*
+ * Update database statistics with non-transactional stats.
+ */
+#define UPDATE_DATABASE_ANYTIME_STATS(dbentry, counts) \
+ do { \
+ (dbentry)->tuples_returned += (counts)->tuples_returned; \
+ (dbentry)->tuples_fetched += (counts)->tuples_fetched; \
+ (dbentry)->blocks_fetched += (counts)->blocks_fetched; \
+ (dbentry)->blocks_hit += (counts)->blocks_hit; \
+ } while (0)
/*
* Copy stats between relations. This is used for things like REINDEX
@@ -789,6 +801,29 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Helper function to flush non-transactional statistics.
+ */
+static void
+flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry, PgStat_TableCounts *counts,
+ bool anytime_only)
+{
+ TimestampTz t;
+
+ tabentry->numscans += counts->numscans;
+ if (counts->numscans)
+ {
+ t = anytime_only ? GetCurrentTimestamp() : GetCurrentTransactionStopTimestamp();
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+
+ tabentry->tuples_returned += counts->tuples_returned;
+ tabentry->tuples_fetched += counts->tuples_fetched;
+ tabentry->blocks_fetched += counts->blocks_fetched;
+ tabentry->blocks_hit += counts->blocks_hit;
+}
+
/*
* Flush out pending stats for the entry
*
@@ -797,9 +832,17 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
*
* Some of the stats are copied to the corresponding pending database stats
* entry when successfully flushing.
+ *
+ * If anytime_only is true, only non-transactional fields are flushed
+ * (numscans, tuples_returned, tuples_fetched, blocks_fetched, blocks_hit).
+ * Transactional fields remain pending until transaction boundary.
+ *
+ * Some of the stats are copied to the corresponding pending database stats
+ * entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -807,12 +850,13 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
- Assert(!anytime_only);
-
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+ /* this is a partial flush if in anytime only mode */
+ *is_partial = anytime_only;
+
/*
* Ignore entries that didn't accumulate any actual counts, such as
* indexes that were opened by the planner but not used.
@@ -824,19 +868,36 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
- /* add the values to the shared entry. */
tabentry = &shtabstats->stats;
- tabentry->numscans += lstats->counts.numscans;
- if (lstats->counts.numscans)
+ if (anytime_only)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
- if (t > tabentry->lastscan)
- tabentry->lastscan = t;
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, true);
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
+ /*
+ * Clear the flushed fields from pending stats to prevent
+ * double-counting when we flush all fields at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
}
- tabentry->tuples_returned += lstats->counts.tuples_returned;
- tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, false);
+
tabentry->tuples_inserted += lstats->counts.tuples_inserted;
tabentry->tuples_updated += lstats->counts.tuples_updated;
tabentry->tuples_deleted += lstats->counts.tuples_deleted;
@@ -866,9 +927,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
*/
tabentry->ins_since_vacuum += lstats->counts.tuples_inserted;
- tabentry->blocks_fetched += lstats->counts.blocks_fetched;
- tabentry->blocks_hit += lstats->counts.blocks_hit;
-
/* Clamp live_tuples in case of negative delta_live_tuples */
tabentry->live_tuples = Max(tabentry->live_tuples, 0);
/* Likewise for dead_tuples */
@@ -878,13 +936,11 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
/* The entry was successfully flushed, add the same to database stats */
dbentry = pgstat_prep_database_pending(dboid);
- dbentry->tuples_returned += lstats->counts.tuples_returned;
- dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
dbentry->tuples_inserted += lstats->counts.tuples_inserted;
dbentry->tuples_updated += lstats->counts.tuples_updated;
dbentry->tuples_deleted += lstats->counts.tuples_deleted;
- dbentry->blocks_fetched += lstats->counts.blocks_fetched;
- dbentry->blocks_hit += lstats->counts.blocks_hit;
return true;
}
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index c4614817966..43fec86c635 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -116,7 +116,8 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
@@ -126,6 +127,9 @@ pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anyti
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ef856dbf55b..06639198f28 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -21,6 +21,7 @@
#include "utils/backend_status.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/pgstat_kind.h"
#include "utils/relcache.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h" /* for backward compatibility */ /* IWYU pragma: export */
@@ -537,10 +538,11 @@ extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
/*
- * Schedule the next anytime stats update timeout.
+ * Schedule the next anytime stats update timeout and mark that we have
+ * mixed anytime stats pending.
*
* This should be called whenever accumulating statistics that support
- * FLUSH_ANYTIME flushing mode.
+ * FLUSH_ANYTIME or FLUSH_MIXED flushing modes.
*/
#define pgstat_schedule_anytime_update() \
do { \
@@ -703,37 +705,58 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned += (n); \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_hit++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 607f4255268..1a2114aad8a 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -322,8 +322,10 @@ typedef struct PgStat_KindInfo
* that cannot use PgStat_EntryRef->pending.
*
* The anytime_only parameter indicates whether this is an anytime flush.
+ * The is_partial parameter indicates whether this is a partial flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait,
+ bool anytime_only, bool *is_partial);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -757,7 +759,8 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -765,7 +768,8 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -790,7 +794,8 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -858,7 +863,8 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..11e3e57806d 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..aef582e7582 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..47414eb6009 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -156,6 +162,8 @@ step s2_func_stats {
}
# Relation stats steps
+step s2_track_counts_on { SET track_counts = on; }
+step s2_track_counts_off { SET track_counts = off; }
step s2_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; }
step s2_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';}
@@ -435,6 +443,23 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_off
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_on
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 207e841911b..ffcda7b6c7a 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -84,7 +84,8 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait, bool anytime_only);
+ bool nowait, bool anytime_only,
+ bool *is_partial);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -152,7 +153,8 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
@@ -160,6 +162,9 @@ test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
shared_entry = (PgStatShared_CustomVarEntry *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-18 11:37 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-02-19 07:06 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Jakub Wartak @ 2026-02-18 11:37 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Wed, Feb 18, 2026 at 6:41 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Tue, Feb 17, 2026 at 01:18:35PM -0600, Sami Imseih wrote:
> > >
> > > > I do not have any further comments on this patchset.
> > >
> > > Thanks for the review!
> >
> > I flipped this CF entry to Ready-for-committer
>
> Thanks!
>
> PFA a mandatory rebase (nothing that needs review) due to a92b809f9da1.
Hi Bertrand!
Thanks for working on this. I've took a quick look on this patchset:
v8-0005: you start using pgstat_schedule_anytime_update() from really
hot macros like pgstat_count_buffer_hit() / pgstat_count_buffer_read()
or pgstat_count_heap_getnext(), e.g.:
#define pgstat_count_buffer_hit(rel)
do {
if (pgstat_should_count_relation(rel))
+ {
(rel)->pgstat_info->counts.blocks_hit++;
+ pgstat_schedule_anytime_update();
+ }
} while (0)
where #define pgstat_schedule_anytime_update() is
do {
if (IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
pgstat_flush_interval);
} while (0)
however that function (get_timeout_active()) is not static inlined so I'm
wondering wouldn't there some major performance impact? Those buffer
macros seem to be pretty heavy hitters, e.g. quite often even per single
buffer in PinBufferForBlock():
pgstat_count_buffer_read(rel);
if (*foundPtr)
pgstat_count_buffer_hit(rel);
so it seems to be:
- often unnecessary double work (and probably as this is a function call to
get_timeout_active it won't be optimized by compiler?)
- but the main question is: why do we need that often to recheck and re-enable
timers so often from such hot places?
v8-0001: this patch modifies ProcessInterrupts() which checks for
AnytimeStatsUpdateTimeoutPending and it may happen that it takes LWLocks
(pgstat_report_anytime_stat()->pgstat_flush_pending_entries()->e.g.
pgstat_*_flush_cb()-> pgstat_lock_entry() -> LWLock)
(It's more a question than finding): isn't it too risky to take that
LWLock from potentially random spots as the CHECK_FOR_INTERRUPTS() is
literally everywhere (~318 places). Wouldn't it be safer to flush
from a couple of desired places?
-J.
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-18 11:37 ` Re: Flush some statistics within running transactions Jakub Wartak <jakub.wartak@enterprisedb.com>
@ 2026-02-19 07:06 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-19 07:06 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi Jakub!
On Wed, Feb 18, 2026 at 12:37:01PM +0100, Jakub Wartak wrote:
> On Wed, Feb 18, 2026 at 6:41 AM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
>
> Thanks for working on this. I've took a quick look on this patchset:
Thanks!
> however that function (get_timeout_active()) is not static inlined so I'm
> wondering wouldn't there some major performance impact?
The get_timeout_active() function is very simple but you are right to be
concerned by the extra function call in hot paths.
So, with a simple pgbench test as:
pgbench -i -s 1
pgbench -n -c1 -j1 -f <(echo "SELECT count(*) FROM pgbench_accounts;") -T 60
1/ with the patch
tps average over 5 runs is about 123.8 and a perf report reports:
0.44% postgres postgres [.] get_timeout_active
0.43% <blabla>;ExecAgg;fetch_input_tuple;IndexOnlyNext;get_timeout_active
To compare with other solutions that will not make use of a get_timeout_active()
function call, let's also check the IndexOnlyNext profile (see above as to why):
12.05% postgres postgres [.] IndexOnlyNext
2/ without the patch
tps average over 5 runs is about 124.9 and a perf report reports:
11.59% postgres postgres [.] IndexOnlyNext
3/ with the patch and get_timeout_active() inlined:
tps average over 5 runs is about 124 and a perf report reports:
10.89% postgres postgres [.] IndexOnlyNext
4/ with the patch and using a boolean instead of get_timeout_active()
tps average over 5 runs is about 129 and a perf report reports:
11.88% postgres postgres [.] IndexOnlyNext
I'm not 100% sure what to conclude here.
What I can see is that get_timeout_active() was about 0.45%, that's not a lot
but that is simple enough to remove that I think that we just should.
> - but the main question is: why do we need that often to recheck and re-enable
> timers so often from such hot places?
We need to ensure that anytime stats are flushed each time we update an anytime
stats. Kind of the same idea as with the existing "pgstat_report_fixed".
> v8-0001: this patch modifies ProcessInterrupts() which checks for
> AnytimeStatsUpdateTimeoutPending and it may happen that it takes LWLocks
> (pgstat_report_anytime_stat()->pgstat_flush_pending_entries()->e.g.
> pgstat_*_flush_cb()-> pgstat_lock_entry() -> LWLock)
>
> (It's more a question than finding): isn't it too risky to take that
> LWLock from potentially random spots as the CHECK_FOR_INTERRUPTS() is
> literally everywhere (~318 places). Wouldn't it be safer to flush
> from a couple of desired places?
That's a good question. pgstat_report_stat() is already called in ProcessInterrupts()
and pgstat_report_anytime_stat() does not do more than it, so I'm tempted to say
that we are fine here.
PFA, a new version that:
- Implements the boolean (i.e "pgstat_pending_anytime") usage instead of the
get_timeout_active() call.
- Fix a race in assign_stats_flush_interval() reported by Sami off thread.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v9-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (42.8K, ../../aZa2ZQ+xLeKQsBgB@ip-10-97-1-34.eu-west-3.compute.internal/2-v9-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From 1314c8096213e95807f45f1c54185f9329cb7dd3 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v9 1/5] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces
spikes when flushed.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducin
a new callback).
- Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush,
relying on PGSTAT_MIN_INTERVAL
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 6 +
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/replication/walsender.c | 8 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 120 +++++++++++++++---
src/backend/utils/activity/pgstat_backend.c | 13 +-
src/backend/utils/activity/pgstat_bgwriter.c | 2 +-
.../utils/activity/pgstat_checkpointer.c | 2 +-
src/backend/utils/activity/pgstat_database.c | 2 +-
src/backend/utils/activity/pgstat_function.c | 4 +-
src/backend/utils/activity/pgstat_io.c | 10 +-
src/backend/utils/activity/pgstat_relation.c | 12 +-
src/backend/utils/activity/pgstat_slru.c | 6 +-
.../utils/activity/pgstat_subscription.c | 4 +-
src/backend/utils/activity/pgstat_wal.c | 10 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 22 ++++
src/include/utils/pgstat_internal.h | 52 ++++++--
src/include/utils/timeout.h | 1 +
.../test_custom_stats/test_custom_var_stats.c | 4 +-
src/tools/pgindent/typedefs.list | 1 +
28 files changed, 278 insertions(+), 66 deletions(-)
10.5% src/backend/postmaster/
5.8% src/backend/replication/
50.9% src/backend/utils/activity/
5.9% src/backend/
18.8% src/include/utils/
6.6% src/include/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..d01b11c7470 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0956bd39a85..059c601c3b8 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..e11c4b099c8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..4954fe425b7 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 742137edad6..f1bae9d23d6 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN); /* no query to cancel */
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 7c0e2809c17..bcf59227a00 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN); /* no query to cancel */
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 10e64a7d1f4..11b7c114d3b 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2cde8ebc729..a7214d0dc6f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
@@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 21de158adbb..2089de782d5 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3564,6 +3564,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..419dc512d9b 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -112,6 +112,7 @@
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +123,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -218,6 +218,12 @@ PgStat_LocalState pgStatLocal;
*/
bool pgstat_report_fixed = false;
+/*
+ * Track when there is pending anytime flush to avoid relying on
+ * get_timeout_active() in hot pathes.
+ */
+bool pgstat_pending_anytime = false;
+
/* ----------
* Local data
*
@@ -288,6 +294,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +312,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +329,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +345,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +363,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +381,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -436,6 +448,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +466,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +484,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +790,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,7 +1296,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
@@ -1345,9 +1349,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,8 +1386,22 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
@@ -1402,6 +1425,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait, anytime_only);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2169,33 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+
+ pgstat_pending_anytime = false;
+}
+
+/*
+ * Timeout handler for flushing anytime stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index f2f8d3ff75f..b09316d3ab3 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
* if some statistics could not be flushed due to lock contention.
*/
bool
-pgstat_flush_backend(bool nowait, bits32 flags)
+pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only)
{
PgStat_EntryRef *entry_ref;
bool has_pending_data = false;
@@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
* If some stats could not be flushed due to lock contention, return true.
*/
bool
-pgstat_backend_flush_cb(bool nowait)
+pgstat_backend_flush_cb(bool nowait, bool anytime_only)
{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
+ return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only);
}
/*
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index ed2fd801189..1c5f0c3ec40 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -61,7 +61,7 @@ pgstat_report_bgwriter(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 1f70194b7a7..2d89a082464 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -68,7 +68,7 @@ pgstat_report_checkpointer(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 933dcb5cae5..8e86df60461 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index e6b84283c6c..5ba4958382f 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
+ Assert(!anytime_only);
+
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..7cd32900236 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void)
* Simpler wrapper of pgstat_io_flush_cb()
*/
void
-pgstat_flush_io(bool nowait)
+pgstat_flush_io(bool nowait, bool anytime_only)
{
- (void) pgstat_io_flush_cb(nowait);
+ (void) pgstat_io_flush_cb(nowait, anytime_only);
}
/*
@@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait)
* acquired. Otherwise, return false.
*/
bool
-pgstat_io_flush_cb(bool nowait)
+pgstat_io_flush_cb(bool nowait, bool anytime_only)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..04d21483d93 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* is done -- which will likely vacuum many relations -- or until the
* VACUUM command has processed all tables and committed.
*/
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel,
pgstat_unlock_entry(entry_ref);
/* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
* entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
+ Assert(!anytime_only);
+
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..bf8a4d58673 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name)
* acquired. Otherwise return false.
*/
bool
-pgstat_slru_flush_cb(bool nowait)
+pgstat_slru_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
int i;
@@ -223,6 +224,9 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 500b1899188..c4614817966 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
+ Assert(!anytime_only);
+
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..2c2f3f10e10 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -51,12 +51,12 @@ pgstat_report_wal(bool force)
nowait = !force;
/* flush wal stats */
- (void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
+ (void) pgstat_wal_flush_cb(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
/* flush IO stats */
- pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -88,7 +88,7 @@ pgstat_wal_have_pending(void)
* acquired. Otherwise return false.
*/
bool
-pgstat_wal_flush_cb(bool nowait)
+pgstat_wal_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
WalUsage wal_usage_diff = {0};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..eeeac1bf39a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..84e698da214 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..f0f546d419a 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,8 +536,24 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
+/*
+ * Schedule the next anytime stats update timeout.
+ *
+ * This should be called whenever accumulating statistics that support
+ * FLUSH_ANYTIME flushing mode.
+ */
+#define pgstat_schedule_anytime_update() \
+ do { \
+ if (IsUnderPostmaster && !pgstat_pending_anytime) \
+ { \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ pgstat_pending_anytime = true; \
+ } \
+ } while (0)
+
extern void pgstat_reset_counters(void);
extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_reset_of_kind(PgStat_Kind kind);
@@ -808,6 +827,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
* Variables in pgstat.c
*/
+extern PGDLLIMPORT bool pgstat_pending_anytime;
+
/* GUC parameters */
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
@@ -851,4 +872,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime;
/* updated by the traffic cop and in errfinish() */
extern PGDLLIMPORT SessionEndType pgStatSessionEndCause;
+
#endif /* PGSTAT_H */
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..607f4255268 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * The mode of when to flush stats. See PgStat_FlushMode for more details.
+ *
+ * This member only has meaning for statistics kinds that accumulate
+ * pending stats and use flush callbacks. For kinds that write directly to
+ * shared memory (e.g., archiver, bgwriter, checkpointer), this member has
+ * no effect.
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo
*
* "pgstat_report_fixed" needs to be set to trigger the flush of pending
* stats.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_static_cb) (bool nowait);
+ bool (*flush_static_cb) (bool nowait, bool anytime_only);
/*
* For fixed-numbered statistics: Reset All.
@@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
@@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
-extern bool pgstat_flush_backend(bool nowait, bits32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
+extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only);
+extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_io.c
*/
-extern void pgstat_flush_io(bool nowait);
+extern void pgstat_flush_io(bool nowait, bool anytime_only);
-extern bool pgstat_io_flush_cb(bool nowait);
+extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
* Functions in pgstat_slru.c
*/
-extern bool pgstat_slru_flush_cb(bool nowait);
+extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void);
*/
extern void pgstat_wal_init_backend_cb(void);
-extern bool pgstat_wal_flush_cb(bool nowait);
+extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
@@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 64a8fe63cce..bc0b5d6e0eb 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait);
+ bool nowait, bool anytime_only);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -150,7 +150,7 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1dbc4b96f51 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2271,6 +2271,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v9-0002-Add-anytime-flush-tests-for-custom-stats.patch (9.0K, ../../aZa2ZQ+xLeKQsBgB@ip-10-97-1-34.eu-west-3.compute.internal/3-v9-0002-Add-anytime-flush-tests-for-custom-stats.patch)
download | inline diff:
From 0199255c67c2ab8b63418c1c5ccef5b6aa1396f2 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 5 Feb 2026 05:54:34 +0000
Subject: [PATCH v9 2/5] Add anytime flush tests for custom stats
---
.../test_custom_stats/t/001_custom_stats.pl | 41 +++++++++++++
.../test_custom_fixed_stats--1.0.sql | 5 ++
.../test_custom_fixed_stats.c | 57 +++++++++++++++++++
.../test_custom_var_stats--1.0.sql | 5 ++
.../test_custom_stats/test_custom_var_stats.c | 27 +++++++++
5 files changed, 135 insertions(+)
33.8% src/test/modules/test_custom_stats/t/
66.1% src/test/modules/test_custom_stats/
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 9e6a7a38577..7be1b281776 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -156,5 +156,46 @@ $result = $node->safe_psql('postgres',
);
is($result, "0", "report of fixed-sized after manual reset");
+# Test FLUSH_ANYTIME mechanism with custom fixed stats
+# This verifies that custom stats can be flushed during a transaction
+
+# Reset stats first
+$node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+my $anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select 'anytime:'||numcalls from test_custom_stats_fixed_report();
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^anytime:2/m,
+ "anytime fixed stats flushed during transaction");
+
+# Test FLUSH_ANYTIME mechanism with custom variable stats
+# This verifies that custom stats can be flushed during a transaction
+
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+$anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_var_anytime_update('entry2');
+ select test_custom_stats_var_anytime_update('entry2');
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select * from test_custom_stats_var_report('entry2');
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^entry2|2|/m,
+ "anytime var stats flushed during transaction");
+
# Test completed successfully
done_testing();
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
index 69a93b5241f..da3a798f289 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
@@ -18,3 +18,8 @@ CREATE FUNCTION test_custom_stats_fixed_reset()
RETURNS void
AS 'MODULE_PATHNAME', 'test_custom_stats_fixed_reset'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_fixed_anytime_update()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
index 908bd18a7c7..30b0fbcbdc7 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
@@ -18,6 +18,7 @@
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_fixed_stats",
@@ -43,11 +44,13 @@ typedef struct PgStatShared_CustomFixedEntry
static void test_custom_stats_fixed_init_shmem_cb(void *stats);
static void test_custom_stats_fixed_reset_all_cb(TimestampTz ts);
static void test_custom_stats_fixed_snapshot_cb(void);
+static bool test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only);
static const PgStat_KindInfo custom_stats = {
.name = "test_custom_fixed_stats",
.fixed_amount = true, /* exactly one entry */
.write_to_file = true, /* persist to stats file */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.shared_size = sizeof(PgStat_StatCustomFixedEntry),
.shared_data_off = offsetof(PgStatShared_CustomFixedEntry, stats),
@@ -56,8 +59,12 @@ static const PgStat_KindInfo custom_stats = {
.init_shmem_cb = test_custom_stats_fixed_init_shmem_cb,
.reset_all_cb = test_custom_stats_fixed_reset_all_cb,
.snapshot_cb = test_custom_stats_fixed_snapshot_cb,
+ .flush_static_cb = test_custom_stats_fixed_flush_cb,
};
+/* Pending statistics */
+static PgStat_StatCustomFixedEntry PendingCustomStats = {0};
+
/*
* Kind ID for test_custom_fixed_stats.
*/
@@ -141,6 +148,38 @@ test_custom_stats_fixed_snapshot_cb(void)
#undef FIXED_COMP
}
+/*
+ * test_custom_stats_fixed_flush_cb
+ * Flush pending stats to shared memory
+ */
+static bool
+test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only)
+{
+ PgStatShared_CustomFixedEntry *stats_shmem;
+
+ /* Nothing to flush if no calls were made */
+ if (PendingCustomStats.numcalls == 0)
+ return false;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_TEST_CUSTOM_FIXED_STATS);
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numcalls += PendingCustomStats.numcalls;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ LWLockRelease(&stats_shmem->lock);
+
+ /* Reset pending stats */
+ PendingCustomStats.numcalls = 0;
+
+ return false; /* successfully flushed */
+}
+
/*--------------------------------------------------------------------------
* SQL-callable functions
*--------------------------------------------------------------------------
@@ -222,3 +261,21 @@ test_custom_stats_fixed_report(PG_FUNCTION_ARGS)
/* Return as tuple */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/*
+ * test_custom_stats_fixed_anytime_update
+ * Increment call counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_fixed_anytime_update);
+Datum
+test_custom_stats_fixed_anytime_update(PG_FUNCTION_ARGS)
+{
+ /* Accumulate in pending stats */
+ PendingCustomStats.numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+ pgstat_report_fixed = true;
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
index 5ed8cfc2dcf..ed66d38981e 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
@@ -24,3 +24,8 @@ CREATE FUNCTION test_custom_stats_var_report(INOUT name TEXT,
RETURNS SETOF record
AS 'MODULE_PATHNAME', 'test_custom_stats_var_report'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_var_anytime_update(IN name TEXT)
+RETURNS void
+AS 'MODULE_PATHNAME', 'test_custom_stats_var_anytime_update'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index bc0b5d6e0eb..207e841911b 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -17,6 +17,7 @@
#include "storage/dsm_registry.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_var_stats",
@@ -107,6 +108,7 @@ static const PgStat_KindInfo custom_stats = {
.name = "test_custom_var_stats",
.fixed_amount = false, /* variable number of entries */
.write_to_file = true, /* persist across restarts */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.track_entry_count = true, /* count active entries */
.accessed_across_databases = true, /* global statistics */
.shared_size = sizeof(PgStatShared_CustomVarEntry),
@@ -689,3 +691,28 @@ test_custom_stats_var_report(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * test_custom_stats_var_anytime_update
+ * Increment custom statistic counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_var_anytime_update);
+Datum
+test_custom_stats_var_anytime_update(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_EntryRef *entry_ref;
+ PgStat_StatCustomVarEntry *pending_entry;
+
+ /* Get pending entry in local memory */
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, InvalidOid,
+ PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name), NULL);
+
+ pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
+ pending_entry->numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+
+ PG_RETURN_VOID();
+}
--
2.34.1
[text/x-diff] v9-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch (9.8K, ../../aZa2ZQ+xLeKQsBgB@ip-10-97-1-34.eu-west-3.compute.internal/4-v9-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch)
download | inline diff:
From 5784c8acff4d51ccd2509be0cd72b760c5f45fa7 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v9 3/5] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 32 +++++++++++++++++++
src/backend/utils/activity/pgstat.c | 13 ++++++++
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/backend/utils/misc/timeout.c | 6 ++++
src/include/pgstat.h | 6 ++--
src/include/utils/guc_hooks.h | 1 +
src/include/utils/timeout.h | 1 +
.../test_custom_stats/t/001_custom_stats.pl | 6 ++--
9 files changed, 70 insertions(+), 6 deletions(-)
51.0% doc/src/sgml/
10.6% src/backend/utils/activity/
15.9% src/backend/utils/misc/
3.6% src/include/utils/
9.0% src/include/
9.6% src/test/modules/test_custom_stats/t/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index faf0bdb62aa..03875b490b7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8932,6 +8932,38 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which certain statistics, which can be updated while a
+ transaction is in progress, are made visible. These include WAL activity
+ and I/O operations.
+ Such statistics are refreshed at the specified interval and can be observed
+ during active transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>
+ and
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is generally
+ the smallest practical value for long-running transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect statistics that are only reported at
+ transaction end, such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics are always
+ flushed at the end of a transaction.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 419dc512d9b..578c575dbfb 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -123,6 +123,8 @@
* ----------
*/
+/* minimum interval non-forced stats flushes.*/
+#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -203,6 +205,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -2170,6 +2173,16 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_all_timeouts_initialized())
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 271c033952e..d2734caafea 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2801,6 +2801,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f938cc65a3a..8bd37a25b38 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -688,6 +688,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/backend/utils/misc/timeout.c b/src/backend/utils/misc/timeout.c
index ddba5dc607c..85c4260d1db 100644
--- a/src/backend/utils/misc/timeout.c
+++ b/src/backend/utils/misc/timeout.c
@@ -828,3 +828,9 @@ get_timeout_finish_time(TimeoutId id)
{
return all_timeouts[id].fin_time;
}
+
+bool
+get_all_timeouts_initialized(void)
+{
+ return all_timeouts_initialized;
+}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f0f546d419a..7829c563316 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,9 +35,6 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
-/* Minimum interval non-forced stats flushes */
-#define PGSTAT_MIN_INTERVAL 1000
-
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -549,7 +546,7 @@ extern void pgstat_force_next_flush(void);
do { \
if (IsUnderPostmaster && !pgstat_pending_anytime) \
{ \
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval); \
pgstat_pending_anytime = true; \
} \
} while (0)
@@ -833,6 +830,7 @@ extern PGDLLIMPORT bool pgstat_pending_anytime;
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 9c90670d9b8..9b5d2a90387 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 10723bb664c..fe7327de209 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -93,5 +93,6 @@ extern bool get_timeout_active(TimeoutId id);
extern bool get_timeout_indicator(TimeoutId id, bool reset_indicator);
extern TimestampTz get_timeout_start_time(TimeoutId id);
extern TimestampTz get_timeout_finish_time(TimeoutId id);
+extern bool get_all_timeouts_initialized(void);
#endif /* TIMEOUT_H */
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 7be1b281776..22e2a75dcb9 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -164,10 +164,11 @@ $node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
my $anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
-- Accumulate stats
select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select 'anytime:'||numcalls from test_custom_stats_fixed_report();
@@ -183,11 +184,12 @@ like($result, qr/^anytime:2/m,
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
$anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
-- Accumulate stats
select test_custom_stats_var_anytime_update('entry2');
select test_custom_stats_var_anytime_update('entry2');
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select * from test_custom_stats_var_report('entry2');
--
2.34.1
[text/x-diff] v9-0004-Remove-useless-calls-to-flush-some-stats.patch (7.6K, ../../aZa2ZQ+xLeKQsBgB@ip-10-97-1-34.eu-west-3.compute.internal/5-v9-0004-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From 4dd666f49b3b970e2520fe82b06d701ea37ef84b Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v9 4/5] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v9-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch (34.2K, ../../aZa2ZQ+xLeKQsBgB@ip-10-97-1-34.eu-west-3.compute.internal/6-v9-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch)
download | inline diff:
From c7a578378f8b70603a12e58d83cc3c6259006245 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v9 5/5] Change RELATION and DATABASE stats to anytime flush
This commit allows mixing fields with different transaction behavior within
the same RELATION or DATABASE statistics kind: some fields are transactional
(e.g., tuple inserts/updates/deletes) while others are non-transactional
(e.g., sequential scans, blocks read).
It modifies the relation flush callback to handle the anytime_only parameter
introduced in commit <nnnn>.
Implementation details:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Change DATABASE from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Add a is_partial parameter to flush_pending_cb() to be able to distinguish
partial flushes in pgstat_flush_pending_entries()
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats. When set to true, it clears flushed fields from pending stats to
prevent double-counting at transaction boundary
DATABASE stats inherit the anytime flush behavior so that relation-derived
stats (tuples_returned, tuples_fetched, blocks_fetched, blocks_hit) are
visible while transactions are in progress.
Tests are added to verify the anytime flush behavior for mixed fields.
---
doc/src/sgml/monitoring.sgml | 37 ++++++-
src/backend/utils/activity/pgstat.c | 15 +--
src/backend/utils/activity/pgstat_database.c | 6 +-
src/backend/utils/activity/pgstat_function.c | 6 +-
src/backend/utils/activity/pgstat_relation.c | 92 ++++++++++++----
.../utils/activity/pgstat_subscription.c | 6 +-
src/include/pgstat.h | 27 ++++-
src/include/utils/pgstat_internal.h | 16 ++-
src/test/isolation/expected/stats.out | 102 ++++++++++++++++++
src/test/isolation/expected/stats_1.out | 102 ++++++++++++++++++
src/test/isolation/specs/stats.spec | 27 ++++-
.../test_custom_stats/test_custom_var_stats.c | 9 +-
12 files changed, 404 insertions(+), 41 deletions(-)
11.7% doc/src/sgml/
26.8% src/backend/utils/activity/
4.2% src/include/utils/
5.4% src/include/
45.1% src/test/isolation/expected/
4.7% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..f2321b631b0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -3956,8 +3969,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_seq_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last sequential scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last sequential scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -3984,8 +3997,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_idx_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last index scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last index scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 578c575dbfb..273258a965a 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -297,7 +297,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -315,7 +315,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -1353,7 +1353,8 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries. For entries
+ * that support it, the callback may flush only non-transactional fields.
* This is safe to call inside transactions.
*
* If anytime_only is false, flushes all entries.
@@ -1384,6 +1385,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1404,7 +1406,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
}
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait,
+ anytime_only, &is_partial_flush);
Assert(did_flush || nowait);
@@ -1414,8 +1417,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 8e86df60461..59dd0790fd7 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,8 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
@@ -443,6 +444,9 @@ pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
pendingent = (PgStat_StatDBEntry *) entry_ref->pending;
sharedent = (PgStatShared_Database *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index 5ba4958382f..44193c93fc7 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,7 +190,8 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
@@ -200,6 +201,9 @@ pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ae2952cae89..62363dacfe1 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -47,7 +47,19 @@ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_lev
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
+static void flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry,
+ PgStat_TableCounts *counts, bool anytime_only);
+/*
+ * Update database statistics with non-transactional stats.
+ */
+#define UPDATE_DATABASE_ANYTIME_STATS(dbentry, counts) \
+ do { \
+ (dbentry)->tuples_returned += (counts)->tuples_returned; \
+ (dbentry)->tuples_fetched += (counts)->tuples_fetched; \
+ (dbentry)->blocks_fetched += (counts)->blocks_fetched; \
+ (dbentry)->blocks_hit += (counts)->blocks_hit; \
+ } while (0)
/*
* Copy stats between relations. This is used for things like REINDEX
@@ -789,6 +801,29 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Helper function to flush non-transactional statistics.
+ */
+static void
+flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry, PgStat_TableCounts *counts,
+ bool anytime_only)
+{
+ TimestampTz t;
+
+ tabentry->numscans += counts->numscans;
+ if (counts->numscans)
+ {
+ t = anytime_only ? GetCurrentTimestamp() : GetCurrentTransactionStopTimestamp();
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+
+ tabentry->tuples_returned += counts->tuples_returned;
+ tabentry->tuples_fetched += counts->tuples_fetched;
+ tabentry->blocks_fetched += counts->blocks_fetched;
+ tabentry->blocks_hit += counts->blocks_hit;
+}
+
/*
* Flush out pending stats for the entry
*
@@ -797,9 +832,17 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
*
* Some of the stats are copied to the corresponding pending database stats
* entry when successfully flushing.
+ *
+ * If anytime_only is true, only non-transactional fields are flushed
+ * (numscans, tuples_returned, tuples_fetched, blocks_fetched, blocks_hit).
+ * Transactional fields remain pending until transaction boundary.
+ *
+ * Some of the stats are copied to the corresponding pending database stats
+ * entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -807,12 +850,13 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
- Assert(!anytime_only);
-
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+ /* this is a partial flush if in anytime only mode */
+ *is_partial = anytime_only;
+
/*
* Ignore entries that didn't accumulate any actual counts, such as
* indexes that were opened by the planner but not used.
@@ -824,19 +868,36 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
- /* add the values to the shared entry. */
tabentry = &shtabstats->stats;
- tabentry->numscans += lstats->counts.numscans;
- if (lstats->counts.numscans)
+ if (anytime_only)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
- if (t > tabentry->lastscan)
- tabentry->lastscan = t;
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, true);
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
+ /*
+ * Clear the flushed fields from pending stats to prevent
+ * double-counting when we flush all fields at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
}
- tabentry->tuples_returned += lstats->counts.tuples_returned;
- tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, false);
+
tabentry->tuples_inserted += lstats->counts.tuples_inserted;
tabentry->tuples_updated += lstats->counts.tuples_updated;
tabentry->tuples_deleted += lstats->counts.tuples_deleted;
@@ -866,9 +927,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
*/
tabentry->ins_since_vacuum += lstats->counts.tuples_inserted;
- tabentry->blocks_fetched += lstats->counts.blocks_fetched;
- tabentry->blocks_hit += lstats->counts.blocks_hit;
-
/* Clamp live_tuples in case of negative delta_live_tuples */
tabentry->live_tuples = Max(tabentry->live_tuples, 0);
/* Likewise for dead_tuples */
@@ -878,13 +936,11 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
/* The entry was successfully flushed, add the same to database stats */
dbentry = pgstat_prep_database_pending(dboid);
- dbentry->tuples_returned += lstats->counts.tuples_returned;
- dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
dbentry->tuples_inserted += lstats->counts.tuples_inserted;
dbentry->tuples_updated += lstats->counts.tuples_updated;
dbentry->tuples_deleted += lstats->counts.tuples_deleted;
- dbentry->blocks_fetched += lstats->counts.blocks_fetched;
- dbentry->blocks_hit += lstats->counts.blocks_hit;
return true;
}
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index c4614817966..43fec86c635 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -116,7 +116,8 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
@@ -126,6 +127,9 @@ pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anyti
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 7829c563316..87cbb539d7c 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -21,6 +21,7 @@
#include "utils/backend_status.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/pgstat_kind.h"
#include "utils/relcache.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h" /* for backward compatibility */ /* IWYU pragma: export */
@@ -537,10 +538,11 @@ extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
/*
- * Schedule the next anytime stats update timeout.
+ * Schedule the next anytime stats update timeout and mark that we have
+ * mixed anytime stats pending.
*
* This should be called whenever accumulating statistics that support
- * FLUSH_ANYTIME flushing mode.
+ * FLUSH_ANYTIME or FLUSH_MIXED flushing modes.
*/
#define pgstat_schedule_anytime_update() \
do { \
@@ -706,37 +708,58 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned += (n); \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_hit++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 607f4255268..1a2114aad8a 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -322,8 +322,10 @@ typedef struct PgStat_KindInfo
* that cannot use PgStat_EntryRef->pending.
*
* The anytime_only parameter indicates whether this is an anytime flush.
+ * The is_partial parameter indicates whether this is a partial flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait,
+ bool anytime_only, bool *is_partial);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -757,7 +759,8 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -765,7 +768,8 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -790,7 +794,8 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -858,7 +863,8 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..11e3e57806d 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..aef582e7582 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..47414eb6009 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -156,6 +162,8 @@ step s2_func_stats {
}
# Relation stats steps
+step s2_track_counts_on { SET track_counts = on; }
+step s2_track_counts_off { SET track_counts = off; }
step s2_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; }
step s2_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';}
@@ -435,6 +443,23 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_off
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_on
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 207e841911b..ffcda7b6c7a 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -84,7 +84,8 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait, bool anytime_only);
+ bool nowait, bool anytime_only,
+ bool *is_partial);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -152,7 +153,8 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
@@ -160,6 +162,9 @@ test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
shared_entry = (PgStatShared_CustomVarEntry *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-19 03:58 ` Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Michael Paquier @ 2026-02-19 03:58 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Wed, Feb 18, 2026 at 05:40:46AM +0000, Bertrand Drouvot wrote:
> PFA a mandatory rebase (nothing that needs review) due to a92b809f9da1.
I don't find the design of this patch appealing, and my mind points
towards two pieces of it:
1) The new requirement related to pgstat_schedule_anytime_update()
that a stats kind needs to call to enable a timeout. This partially
doubles with pgstat_report_fixed. And I suspect that this extra set
of requirements, introducing a new level of complexity for in-core
stats kinds as well as extension developers, would be the source of
more bugs.
2) The timeout requirement itself, relying on a timeout threshold
controlled by a backend-side configuration.
With that in mind, wouldn't it be simpler if we introduced an API that
could be used from client applications instead, in a model similar
what we do for procsignal.c/h? One such example is
LOG_MEMORY_CONTEXT, where we have a SQL function that is able to tell
to a backend that it needs to do something. I could see various
benefits to this approach, because it gives more flexibility with the
timing of the stats flushes, which may not be a backend-side only
policy:
- Use a cron bgworker in the backend, that scans pg_stat_activity, for
example for long-running transactions based on a threshold.
- Do the same periodic scan of pg_stat_activity, but from a client
application.
The PROCSIG would need to set a flag in a new SIGUSR1 handler that
would trigger the flush for the stats kinds that have the
out-of-transaction property set once we go through in
ProcessInterrupts(). We already have a pgstats report call there,
hence it is a matter of removing the timeout requirements as presented
in the patch, and let client applications when this should happen.
The property of tracking which stats kind is surely important, Sami
has reminded that a couple of hours ago that there are some stats that
we should not flush even if we get an async request. Another thing
that I am doubting about is if using the same async flush threshold
makes sense for everything. Long-running transactions, for example,
mostly would not care much even if we use an interval less aggressive
than what a WAL sender sees.
Not a fan of the hardcoded sleeps in the tests, either. On fast
machines, these tend to waste in runtime because a process stands idle
doing nothing. On slow machines, tests could be unstable if a sleep
takes longer than it takes for the environment to react to a condition
of the test.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aZaKVOrRO_bHsevn@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
@ 2026-02-19 08:01 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 23:48 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-19 08:01 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Thu, Feb 19, 2026 at 12:58:12PM +0900, Michael Paquier wrote:
> On Wed, Feb 18, 2026 at 05:40:46AM +0000, Bertrand Drouvot wrote:
> > PFA a mandatory rebase (nothing that needs review) due to a92b809f9da1.
>
> I don't find the design of this patch appealing, and my mind points
> towards two pieces of it:
> 1) The new requirement related to pgstat_schedule_anytime_update()
> that a stats kind needs to call to enable a timeout. This partially
> doubles with pgstat_report_fixed. And I suspect that this extra set
> of requirements, introducing a new level of complexity for in-core
> stats kinds as well as extension developers, would be the source of
> more bugs.
Yeah, maybe we should re-think the way we report that we have something to flush,
but I think that's more a general discussion that should also take care of
pgstat_report_fixed.
> 2) The timeout requirement itself, relying on a timeout threshold
> controlled by a backend-side configuration.
What are you concerns with this?
> With that in mind, wouldn't it be simpler if we introduced an API that
> could be used from client applications instead, in a model similar
> what we do for procsignal.c/h?
That's another angle to look at it but I think that giving this responsability to
the clients would not solve the concerns we had in [1] (that led to 039549d70f6
and to this thread). It seems to me that a solution/design that does not allow
us to "revert" 039549d70f6 does not suit our needs. Thoughts?
> Not a fan of the hardcoded sleeps in the tests, either.
Yeah, after our off-list discussion yesterday, I tried to implement the same
trick that f1e251be80a has done with injection points (nice trick by the way!),
but that led to:
TRAP: failed Assert("CritSectionCount == 0 || (context)->allowInCritSection"), File: "mcxt.c", Line: 1237
postgres: main: walwriter (ExceptionalCondition+0x9e)[0xc4ce4d]
postgres: main: walwriter (MemoryContextAlloc+0x8c)[0xc8f0ec]
postgres: main: walwriter (MemoryContextStrdup+0x37)[0xc8fea1]
postgres: main: walwriter (pstrdup+0x22)[0xc8fee4]
postgres: main: walwriter (substitute_path_macro+0x65)[0xc56068]
postgres: main: walwriter [0xc55e90]
postgres: main: walwriter (load_external_function+0x59)[0xc553db]
postgres: main: walwriter [0xc7b858]
postgres: main: walwriter [0xc7c125]
postgres: main: walwriter (IsInjectionPointAttached+0x18)[0xc7c20c]
postgres: main: walwriter (pgstat_count_backend_io_op+0x12f)[0xa9a116]
postgres: main: walwriter (pgstat_count_io_op+0x169)[0xa9cb57]
postgres: main: walwriter (pgstat_count_io_op_time+0x1cc)[0xa9cda7]
So, I did not spend that much time on it. I could if we strongly think that those
sleeps have to be discarded though.
[1]: https://postgr.es/m/erpzwxoptqhuptdrtehqydzjapvroumkhh7lc6poclbhe7jk7l%40l3yfsq5q4pw7
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-19 22:08 ` Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-02-19 22:08 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> > Not a fan of the hardcoded sleeps in the tests, either.
>
> Yeah, after our off-list discussion yesterday, I tried to implement the same
> trick that f1e251be80a has done with injection points (nice trick by the way!),
> but that led to:
>
> TRAP: failed Assert("CritSectionCount == 0 || (context)->allowInCritSection"), File: "mcxt.c", Line: 1237
> postgres: main: walwriter (ExceptionalCondition+0x9e)[0xc4ce4d]
> postgres: main: walwriter (MemoryContextAlloc+0x8c)[0xc8f0ec]
> postgres: main: walwriter (MemoryContextStrdup+0x37)[0xc8fea1]
> postgres: main: walwriter (pstrdup+0x22)[0xc8fee4]
> postgres: main: walwriter (substitute_path_macro+0x65)[0xc56068]
> postgres: main: walwriter [0xc55e90]
> postgres: main: walwriter (load_external_function+0x59)[0xc553db]
> postgres: main: walwriter [0xc7b858]
> postgres: main: walwriter [0xc7c125]
> postgres: main: walwriter (IsInjectionPointAttached+0x18)[0xc7c20c]
> postgres: main: walwriter (pgstat_count_backend_io_op+0x12f)[0xa9a116]
> postgres: main: walwriter (pgstat_count_io_op+0x169)[0xa9cb57]
> postgres: main: walwriter (pgstat_count_io_op_time+0x1cc)[0xa9cda7]
>
> So, I did not spend that much time on it. I could if we strongly think that those
> sleeps have to be discarded though.
I took a look at this today out of interest, based on what you mentioned to me
offline.
There is this in injection_points.c
```
/*
* Load an injection point into the local cache.
*
* This is useful to be able to load an injection point before running it,
* especially if the injection point is called in a code path where memory
* allocations cannot happen, like critical sections.
*/
void
InjectionPointLoad(const char *name)
{
#ifdef USE_INJECTION_POINTS
InjectionPointCacheRefresh(name);
#else
elog(ERROR, "Injection points are not supported by this build");
#endif
}
```
so, instead of calling IS_INJECTION_POINT_ATTACHED macro which is
called on-demand and in the case of I/O stats, during the critical section,
you can just call INJECTION_POINT_LOAD during pgstat_initialize(),
like this:
```
INJECTION_POINT_LOAD("anytime-update-reduce-timeout");
```
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-20 15:55 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-20 15:55 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Thu, Feb 19, 2026 at 04:08:59PM -0600, Sami Imseih wrote:
> > > Not a fan of the hardcoded sleeps in the tests, either.
> >
> > Yeah, after our off-list discussion yesterday, I tried to implement the same
> > trick that f1e251be80a has done with injection points (nice trick by the way!),
PFA, a tiny ("include") rebase due to 9842e8aca09 (the v9 patch series was applying
but not compiling).
> I took a look at this today out of interest,
Thanks!
> so, instead of calling IS_INJECTION_POINT_ATTACHED macro which is
I think that not calling IS_INJECTION_POINT_ATTACHED() but only relying on
"ifdef USE_INJECTION_POINTS" would set the tiny timeout value to the entire test
suit.
I'm waiting for Michael's feedback about the current design ([1]) to see if it's
still worth spending time improving the tests.
[1]: https://postgr.es/m/aZbDYMrOkeCyIubO%40ip-10-97-1-34.eu-west-3.compute.internal
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v10-0001-Add-pgstat_report_anytime_stat-for-periodic-stat.patch (42.9K, ../../aZiD74ZVehxfgqxM@ip-10-97-1-34.eu-west-3.compute.internal/2-v10-0001-Add-pgstat_report_anytime_stat-for-periodic-stat.patch)
download | inline diff:
From 653d6a56427e9789ffa07d03f131563cd75270fc Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v10 1/5] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces
spikes when flushed.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducin
a new callback).
- Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush,
relying on PGSTAT_MIN_INTERVAL
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 6 +
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/replication/walsender.c | 8 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 121 +++++++++++++++---
src/backend/utils/activity/pgstat_backend.c | 13 +-
src/backend/utils/activity/pgstat_bgwriter.c | 2 +-
.../utils/activity/pgstat_checkpointer.c | 2 +-
src/backend/utils/activity/pgstat_database.c | 2 +-
src/backend/utils/activity/pgstat_function.c | 4 +-
src/backend/utils/activity/pgstat_io.c | 10 +-
src/backend/utils/activity/pgstat_relation.c | 12 +-
src/backend/utils/activity/pgstat_slru.c | 6 +-
.../utils/activity/pgstat_subscription.c | 4 +-
src/backend/utils/activity/pgstat_wal.c | 10 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 22 ++++
src/include/utils/pgstat_internal.h | 52 ++++++--
src/include/utils/timeout.h | 1 +
.../test_custom_stats/test_custom_var_stats.c | 4 +-
src/tools/pgindent/typedefs.list | 1 +
28 files changed, 279 insertions(+), 66 deletions(-)
10.5% src/backend/postmaster/
5.8% src/backend/replication/
51.0% src/backend/utils/activity/
5.8% src/backend/
18.7% src/include/utils/
6.6% src/include/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13cce9b49f1..cf29fc91f70 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0956bd39a85..059c601c3b8 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..e11c4b099c8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..4954fe425b7 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 742137edad6..f1bae9d23d6 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN); /* no query to cancel */
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 7c0e2809c17..bcf59227a00 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN); /* no query to cancel */
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 7c1b8757d7d..aecc7a127e6 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2cde8ebc729..a7214d0dc6f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
@@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d01a09dd0c4..8c30efa2443 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3564,6 +3564,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..ddd331e2c81 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -108,10 +108,12 @@
#include "pgstat.h"
#include "storage/fd.h"
#include "storage/ipc.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +124,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -218,6 +219,12 @@ PgStat_LocalState pgStatLocal;
*/
bool pgstat_report_fixed = false;
+/*
+ * Track when there is pending anytime flush to avoid relying on
+ * get_timeout_active() in hot pathes.
+ */
+bool pgstat_pending_anytime = false;
+
/* ----------
* Local data
*
@@ -288,6 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +330,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +346,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +364,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +382,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -436,6 +449,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +467,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +485,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +791,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,7 +1297,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
@@ -1345,9 +1350,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,8 +1387,22 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
@@ -1402,6 +1426,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait, anytime_only);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+
+ pgstat_pending_anytime = false;
+}
+
+/*
+ * Timeout handler for flushing anytime stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index f2f8d3ff75f..b09316d3ab3 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
* if some statistics could not be flushed due to lock contention.
*/
bool
-pgstat_flush_backend(bool nowait, bits32 flags)
+pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only)
{
PgStat_EntryRef *entry_ref;
bool has_pending_data = false;
@@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
* If some stats could not be flushed due to lock contention, return true.
*/
bool
-pgstat_backend_flush_cb(bool nowait)
+pgstat_backend_flush_cb(bool nowait, bool anytime_only)
{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
+ return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only);
}
/*
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index ed2fd801189..1c5f0c3ec40 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -61,7 +61,7 @@ pgstat_report_bgwriter(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 1f70194b7a7..2d89a082464 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -68,7 +68,7 @@ pgstat_report_checkpointer(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 933dcb5cae5..8e86df60461 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index e6b84283c6c..5ba4958382f 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
+ Assert(!anytime_only);
+
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..7cd32900236 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void)
* Simpler wrapper of pgstat_io_flush_cb()
*/
void
-pgstat_flush_io(bool nowait)
+pgstat_flush_io(bool nowait, bool anytime_only)
{
- (void) pgstat_io_flush_cb(nowait);
+ (void) pgstat_io_flush_cb(nowait, anytime_only);
}
/*
@@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait)
* acquired. Otherwise, return false.
*/
bool
-pgstat_io_flush_cb(bool nowait)
+pgstat_io_flush_cb(bool nowait, bool anytime_only)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..04d21483d93 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* is done -- which will likely vacuum many relations -- or until the
* VACUUM command has processed all tables and committed.
*/
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel,
pgstat_unlock_entry(entry_ref);
/* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
* entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
+ Assert(!anytime_only);
+
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..bf8a4d58673 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name)
* acquired. Otherwise return false.
*/
bool
-pgstat_slru_flush_cb(bool nowait)
+pgstat_slru_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
int i;
@@ -223,6 +224,9 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 3277cf88a4e..6b6eec7578d 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -117,11 +117,13 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
+ Assert(!anytime_only);
+
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..2c2f3f10e10 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -51,12 +51,12 @@ pgstat_report_wal(bool force)
nowait = !force;
/* flush wal stats */
- (void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
+ (void) pgstat_wal_flush_cb(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
/* flush IO stats */
- pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -88,7 +88,7 @@ pgstat_wal_have_pending(void)
* acquired. Otherwise return false.
*/
bool
-pgstat_wal_flush_cb(bool nowait)
+pgstat_wal_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
WalUsage wal_usage_diff = {0};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..eeeac1bf39a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..84e698da214 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 9bb777c3d5a..b011a315679 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,6 +34,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -532,8 +535,24 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
+/*
+ * Schedule the next anytime stats update timeout.
+ *
+ * This should be called whenever accumulating statistics that support
+ * FLUSH_ANYTIME flushing mode.
+ */
+#define pgstat_schedule_anytime_update() \
+ do { \
+ if (IsUnderPostmaster && !pgstat_pending_anytime) \
+ { \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ pgstat_pending_anytime = true; \
+ } \
+ } while (0)
+
extern void pgstat_reset_counters(void);
extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_reset_of_kind(PgStat_Kind kind);
@@ -806,6 +825,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
* Variables in pgstat.c
*/
+extern PGDLLIMPORT bool pgstat_pending_anytime;
+
/* GUC parameters */
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
@@ -849,4 +870,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime;
/* updated by the traffic cop and in errfinish() */
extern PGDLLIMPORT SessionEndType pgStatSessionEndCause;
+
#endif /* PGSTAT_H */
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..607f4255268 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * The mode of when to flush stats. See PgStat_FlushMode for more details.
+ *
+ * This member only has meaning for statistics kinds that accumulate
+ * pending stats and use flush callbacks. For kinds that write directly to
+ * shared memory (e.g., archiver, bgwriter, checkpointer), this member has
+ * no effect.
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo
*
* "pgstat_report_fixed" needs to be set to trigger the flush of pending
* stats.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_static_cb) (bool nowait);
+ bool (*flush_static_cb) (bool nowait, bool anytime_only);
/*
* For fixed-numbered statistics: Reset All.
@@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
@@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
-extern bool pgstat_flush_backend(bool nowait, bits32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
+extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only);
+extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_io.c
*/
-extern void pgstat_flush_io(bool nowait);
+extern void pgstat_flush_io(bool nowait, bool anytime_only);
-extern bool pgstat_io_flush_cb(bool nowait);
+extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
* Functions in pgstat_slru.c
*/
-extern bool pgstat_slru_flush_cb(bool nowait);
+extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void);
*/
extern void pgstat_wal_init_backend_cb(void);
-extern bool pgstat_wal_flush_cb(bool nowait);
+extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
@@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index da28afbd929..4c207611236 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -84,7 +84,7 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait);
+ bool nowait, bool anytime_only);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -151,7 +151,7 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1dbc4b96f51 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2271,6 +2271,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v10-0002-Add-anytime-flush-tests-for-custom-stats.patch (9.0K, ../../aZiD74ZVehxfgqxM@ip-10-97-1-34.eu-west-3.compute.internal/3-v10-0002-Add-anytime-flush-tests-for-custom-stats.patch)
download | inline diff:
From cf2dffc645aefb1db5d5e28de2d923e7ffa2f346 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 5 Feb 2026 05:54:34 +0000
Subject: [PATCH v10 2/5] Add anytime flush tests for custom stats
---
.../test_custom_stats/t/001_custom_stats.pl | 41 +++++++++++++
.../test_custom_fixed_stats--1.0.sql | 5 ++
.../test_custom_fixed_stats.c | 57 +++++++++++++++++++
.../test_custom_var_stats--1.0.sql | 5 ++
.../test_custom_stats/test_custom_var_stats.c | 27 +++++++++
5 files changed, 135 insertions(+)
33.8% src/test/modules/test_custom_stats/t/
66.1% src/test/modules/test_custom_stats/
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 9e6a7a38577..7be1b281776 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -156,5 +156,46 @@ $result = $node->safe_psql('postgres',
);
is($result, "0", "report of fixed-sized after manual reset");
+# Test FLUSH_ANYTIME mechanism with custom fixed stats
+# This verifies that custom stats can be flushed during a transaction
+
+# Reset stats first
+$node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+my $anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select 'anytime:'||numcalls from test_custom_stats_fixed_report();
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^anytime:2/m,
+ "anytime fixed stats flushed during transaction");
+
+# Test FLUSH_ANYTIME mechanism with custom variable stats
+# This verifies that custom stats can be flushed during a transaction
+
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+$anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_var_anytime_update('entry2');
+ select test_custom_stats_var_anytime_update('entry2');
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select * from test_custom_stats_var_report('entry2');
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^entry2|2|/m,
+ "anytime var stats flushed during transaction");
+
# Test completed successfully
done_testing();
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
index 69a93b5241f..da3a798f289 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
@@ -18,3 +18,8 @@ CREATE FUNCTION test_custom_stats_fixed_reset()
RETURNS void
AS 'MODULE_PATHNAME', 'test_custom_stats_fixed_reset'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_fixed_anytime_update()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
index 908bd18a7c7..30b0fbcbdc7 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
@@ -18,6 +18,7 @@
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_fixed_stats",
@@ -43,11 +44,13 @@ typedef struct PgStatShared_CustomFixedEntry
static void test_custom_stats_fixed_init_shmem_cb(void *stats);
static void test_custom_stats_fixed_reset_all_cb(TimestampTz ts);
static void test_custom_stats_fixed_snapshot_cb(void);
+static bool test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only);
static const PgStat_KindInfo custom_stats = {
.name = "test_custom_fixed_stats",
.fixed_amount = true, /* exactly one entry */
.write_to_file = true, /* persist to stats file */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.shared_size = sizeof(PgStat_StatCustomFixedEntry),
.shared_data_off = offsetof(PgStatShared_CustomFixedEntry, stats),
@@ -56,8 +59,12 @@ static const PgStat_KindInfo custom_stats = {
.init_shmem_cb = test_custom_stats_fixed_init_shmem_cb,
.reset_all_cb = test_custom_stats_fixed_reset_all_cb,
.snapshot_cb = test_custom_stats_fixed_snapshot_cb,
+ .flush_static_cb = test_custom_stats_fixed_flush_cb,
};
+/* Pending statistics */
+static PgStat_StatCustomFixedEntry PendingCustomStats = {0};
+
/*
* Kind ID for test_custom_fixed_stats.
*/
@@ -141,6 +148,38 @@ test_custom_stats_fixed_snapshot_cb(void)
#undef FIXED_COMP
}
+/*
+ * test_custom_stats_fixed_flush_cb
+ * Flush pending stats to shared memory
+ */
+static bool
+test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only)
+{
+ PgStatShared_CustomFixedEntry *stats_shmem;
+
+ /* Nothing to flush if no calls were made */
+ if (PendingCustomStats.numcalls == 0)
+ return false;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_TEST_CUSTOM_FIXED_STATS);
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numcalls += PendingCustomStats.numcalls;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ LWLockRelease(&stats_shmem->lock);
+
+ /* Reset pending stats */
+ PendingCustomStats.numcalls = 0;
+
+ return false; /* successfully flushed */
+}
+
/*--------------------------------------------------------------------------
* SQL-callable functions
*--------------------------------------------------------------------------
@@ -222,3 +261,21 @@ test_custom_stats_fixed_report(PG_FUNCTION_ARGS)
/* Return as tuple */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/*
+ * test_custom_stats_fixed_anytime_update
+ * Increment call counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_fixed_anytime_update);
+Datum
+test_custom_stats_fixed_anytime_update(PG_FUNCTION_ARGS)
+{
+ /* Accumulate in pending stats */
+ PendingCustomStats.numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+ pgstat_report_fixed = true;
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
index 5ed8cfc2dcf..ed66d38981e 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
@@ -24,3 +24,8 @@ CREATE FUNCTION test_custom_stats_var_report(INOUT name TEXT,
RETURNS SETOF record
AS 'MODULE_PATHNAME', 'test_custom_stats_var_report'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_var_anytime_update(IN name TEXT)
+RETURNS void
+AS 'MODULE_PATHNAME', 'test_custom_stats_var_anytime_update'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 4c207611236..e9f1bda6b32 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -18,6 +18,7 @@
#include "storage/dsm_registry.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_var_stats",
@@ -108,6 +109,7 @@ static const PgStat_KindInfo custom_stats = {
.name = "test_custom_var_stats",
.fixed_amount = false, /* variable number of entries */
.write_to_file = true, /* persist across restarts */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.track_entry_count = true, /* count active entries */
.accessed_across_databases = true, /* global statistics */
.shared_size = sizeof(PgStatShared_CustomVarEntry),
@@ -690,3 +692,28 @@ test_custom_stats_var_report(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * test_custom_stats_var_anytime_update
+ * Increment custom statistic counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_var_anytime_update);
+Datum
+test_custom_stats_var_anytime_update(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_EntryRef *entry_ref;
+ PgStat_StatCustomVarEntry *pending_entry;
+
+ /* Get pending entry in local memory */
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, InvalidOid,
+ PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name), NULL);
+
+ pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
+ pending_entry->numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+
+ PG_RETURN_VOID();
+}
--
2.34.1
[text/x-diff] v10-0003-Add-GUC-to-specify-non-transactional-statistics-.patch (9.8K, ../../aZiD74ZVehxfgqxM@ip-10-97-1-34.eu-west-3.compute.internal/4-v10-0003-Add-GUC-to-specify-non-transactional-statistics-.patch)
download | inline diff:
From ca50ef477a3eb1bf0384f49126c9d95f28438927 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v10 3/5] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 32 +++++++++++++++++++
src/backend/utils/activity/pgstat.c | 13 ++++++++
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/backend/utils/misc/timeout.c | 6 ++++
src/include/pgstat.h | 6 ++--
src/include/utils/guc_hooks.h | 1 +
src/include/utils/timeout.h | 1 +
.../test_custom_stats/t/001_custom_stats.pl | 6 ++--
9 files changed, 70 insertions(+), 6 deletions(-)
51.0% doc/src/sgml/
10.6% src/backend/utils/activity/
15.9% src/backend/utils/misc/
3.6% src/include/utils/
9.0% src/include/
9.6% src/test/modules/test_custom_stats/t/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 20dbcaeb3ee..1eed71007a7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8929,6 +8929,38 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which certain statistics, which can be updated while a
+ transaction is in progress, are made visible. These include WAL activity
+ and I/O operations.
+ Such statistics are refreshed at the specified interval and can be observed
+ during active transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>
+ and
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is generally
+ the smallest practical value for long-running transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect statistics that are only reported at
+ transaction end, such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics are always
+ flushed at the end of a transaction.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ddd331e2c81..fd6ab0db16f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -124,6 +124,8 @@
* ----------
*/
+/* minimum interval non-forced stats flushes.*/
+#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -204,6 +206,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -2171,6 +2174,16 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_all_timeouts_initialized())
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 9507778415d..073e08c7892 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2801,6 +2801,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f938cc65a3a..8bd37a25b38 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -688,6 +688,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/backend/utils/misc/timeout.c b/src/backend/utils/misc/timeout.c
index ddba5dc607c..85c4260d1db 100644
--- a/src/backend/utils/misc/timeout.c
+++ b/src/backend/utils/misc/timeout.c
@@ -828,3 +828,9 @@ get_timeout_finish_time(TimeoutId id)
{
return all_timeouts[id].fin_time;
}
+
+bool
+get_all_timeouts_initialized(void)
+{
+ return all_timeouts_initialized;
+}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b011a315679..90237c70829 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,9 +34,6 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
-/* Minimum interval non-forced stats flushes */
-#define PGSTAT_MIN_INTERVAL 1000
-
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -548,7 +545,7 @@ extern void pgstat_force_next_flush(void);
do { \
if (IsUnderPostmaster && !pgstat_pending_anytime) \
{ \
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval); \
pgstat_pending_anytime = true; \
} \
} while (0)
@@ -831,6 +828,7 @@ extern PGDLLIMPORT bool pgstat_pending_anytime;
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 9c90670d9b8..9b5d2a90387 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 10723bb664c..fe7327de209 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -93,5 +93,6 @@ extern bool get_timeout_active(TimeoutId id);
extern bool get_timeout_indicator(TimeoutId id, bool reset_indicator);
extern TimestampTz get_timeout_start_time(TimeoutId id);
extern TimestampTz get_timeout_finish_time(TimeoutId id);
+extern bool get_all_timeouts_initialized(void);
#endif /* TIMEOUT_H */
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 7be1b281776..22e2a75dcb9 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -164,10 +164,11 @@ $node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
my $anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
-- Accumulate stats
select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select 'anytime:'||numcalls from test_custom_stats_fixed_report();
@@ -183,11 +184,12 @@ like($result, qr/^anytime:2/m,
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
$anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
-- Accumulate stats
select test_custom_stats_var_anytime_update('entry2');
select test_custom_stats_var_anytime_update('entry2');
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select * from test_custom_stats_var_report('entry2');
--
2.34.1
[text/x-diff] v10-0004-Remove-useless-calls-to-flush-some-stats.patch (7.7K, ../../aZiD74ZVehxfgqxM@ip-10-97-1-34.eu-west-3.compute.internal/5-v10-0004-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From d27bf3b7ddb430bc145764db842eac8e41c9b9e8 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v10 4/5] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index aecc7a127e6..edf5ac65660 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v10-0005-Change-RELATION-and-DATABASE-stats-to-anytime-fl.patch (34.2K, ../../aZiD74ZVehxfgqxM@ip-10-97-1-34.eu-west-3.compute.internal/6-v10-0005-Change-RELATION-and-DATABASE-stats-to-anytime-fl.patch)
download | inline diff:
From a9dc58fbd4474adf36fd64de54386cc2311b2484 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v10 5/5] Change RELATION and DATABASE stats to anytime flush
This commit allows mixing fields with different transaction behavior within
the same RELATION or DATABASE statistics kind: some fields are transactional
(e.g., tuple inserts/updates/deletes) while others are non-transactional
(e.g., sequential scans, blocks read).
It modifies the relation flush callback to handle the anytime_only parameter
introduced in commit <nnnn>.
Implementation details:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Change DATABASE from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Add a is_partial parameter to flush_pending_cb() to be able to distinguish
partial flushes in pgstat_flush_pending_entries()
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats. When set to true, it clears flushed fields from pending stats to
prevent double-counting at transaction boundary
DATABASE stats inherit the anytime flush behavior so that relation-derived
stats (tuples_returned, tuples_fetched, blocks_fetched, blocks_hit) are
visible while transactions are in progress.
Tests are added to verify the anytime flush behavior for mixed fields.
---
doc/src/sgml/monitoring.sgml | 37 ++++++-
src/backend/utils/activity/pgstat.c | 15 +--
src/backend/utils/activity/pgstat_database.c | 6 +-
src/backend/utils/activity/pgstat_function.c | 6 +-
src/backend/utils/activity/pgstat_relation.c | 92 ++++++++++++----
.../utils/activity/pgstat_subscription.c | 6 +-
src/include/pgstat.h | 27 ++++-
src/include/utils/pgstat_internal.h | 16 ++-
src/test/isolation/expected/stats.out | 102 ++++++++++++++++++
src/test/isolation/expected/stats_1.out | 102 ++++++++++++++++++
src/test/isolation/specs/stats.spec | 27 ++++-
.../test_custom_stats/test_custom_var_stats.c | 9 +-
12 files changed, 404 insertions(+), 41 deletions(-)
11.7% doc/src/sgml/
26.8% src/backend/utils/activity/
4.2% src/include/utils/
5.4% src/include/
45.1% src/test/isolation/expected/
4.7% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..f2321b631b0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -3956,8 +3969,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_seq_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last sequential scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last sequential scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -3984,8 +3997,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_idx_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last index scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last index scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index fd6ab0db16f..a8a905640d0 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -298,7 +298,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -316,7 +316,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -1354,7 +1354,8 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries. For entries
+ * that support it, the callback may flush only non-transactional fields.
* This is safe to call inside transactions.
*
* If anytime_only is false, flushes all entries.
@@ -1385,6 +1386,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1405,7 +1407,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
}
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait,
+ anytime_only, &is_partial_flush);
Assert(did_flush || nowait);
@@ -1415,8 +1418,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 8e86df60461..59dd0790fd7 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,8 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
@@ -443,6 +444,9 @@ pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
pendingent = (PgStat_StatDBEntry *) entry_ref->pending;
sharedent = (PgStatShared_Database *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index 5ba4958382f..44193c93fc7 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,7 +190,8 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
@@ -200,6 +201,9 @@ pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ae2952cae89..62363dacfe1 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -47,7 +47,19 @@ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_lev
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
+static void flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry,
+ PgStat_TableCounts *counts, bool anytime_only);
+/*
+ * Update database statistics with non-transactional stats.
+ */
+#define UPDATE_DATABASE_ANYTIME_STATS(dbentry, counts) \
+ do { \
+ (dbentry)->tuples_returned += (counts)->tuples_returned; \
+ (dbentry)->tuples_fetched += (counts)->tuples_fetched; \
+ (dbentry)->blocks_fetched += (counts)->blocks_fetched; \
+ (dbentry)->blocks_hit += (counts)->blocks_hit; \
+ } while (0)
/*
* Copy stats between relations. This is used for things like REINDEX
@@ -789,6 +801,29 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Helper function to flush non-transactional statistics.
+ */
+static void
+flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry, PgStat_TableCounts *counts,
+ bool anytime_only)
+{
+ TimestampTz t;
+
+ tabentry->numscans += counts->numscans;
+ if (counts->numscans)
+ {
+ t = anytime_only ? GetCurrentTimestamp() : GetCurrentTransactionStopTimestamp();
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+
+ tabentry->tuples_returned += counts->tuples_returned;
+ tabentry->tuples_fetched += counts->tuples_fetched;
+ tabentry->blocks_fetched += counts->blocks_fetched;
+ tabentry->blocks_hit += counts->blocks_hit;
+}
+
/*
* Flush out pending stats for the entry
*
@@ -797,9 +832,17 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
*
* Some of the stats are copied to the corresponding pending database stats
* entry when successfully flushing.
+ *
+ * If anytime_only is true, only non-transactional fields are flushed
+ * (numscans, tuples_returned, tuples_fetched, blocks_fetched, blocks_hit).
+ * Transactional fields remain pending until transaction boundary.
+ *
+ * Some of the stats are copied to the corresponding pending database stats
+ * entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -807,12 +850,13 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
- Assert(!anytime_only);
-
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+ /* this is a partial flush if in anytime only mode */
+ *is_partial = anytime_only;
+
/*
* Ignore entries that didn't accumulate any actual counts, such as
* indexes that were opened by the planner but not used.
@@ -824,19 +868,36 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
- /* add the values to the shared entry. */
tabentry = &shtabstats->stats;
- tabentry->numscans += lstats->counts.numscans;
- if (lstats->counts.numscans)
+ if (anytime_only)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
- if (t > tabentry->lastscan)
- tabentry->lastscan = t;
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, true);
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
+ /*
+ * Clear the flushed fields from pending stats to prevent
+ * double-counting when we flush all fields at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
}
- tabentry->tuples_returned += lstats->counts.tuples_returned;
- tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, false);
+
tabentry->tuples_inserted += lstats->counts.tuples_inserted;
tabentry->tuples_updated += lstats->counts.tuples_updated;
tabentry->tuples_deleted += lstats->counts.tuples_deleted;
@@ -866,9 +927,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
*/
tabentry->ins_since_vacuum += lstats->counts.tuples_inserted;
- tabentry->blocks_fetched += lstats->counts.blocks_fetched;
- tabentry->blocks_hit += lstats->counts.blocks_hit;
-
/* Clamp live_tuples in case of negative delta_live_tuples */
tabentry->live_tuples = Max(tabentry->live_tuples, 0);
/* Likewise for dead_tuples */
@@ -878,13 +936,11 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
/* The entry was successfully flushed, add the same to database stats */
dbentry = pgstat_prep_database_pending(dboid);
- dbentry->tuples_returned += lstats->counts.tuples_returned;
- dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
dbentry->tuples_inserted += lstats->counts.tuples_inserted;
dbentry->tuples_updated += lstats->counts.tuples_updated;
dbentry->tuples_deleted += lstats->counts.tuples_deleted;
- dbentry->blocks_fetched += lstats->counts.blocks_fetched;
- dbentry->blocks_hit += lstats->counts.blocks_hit;
return true;
}
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 6b6eec7578d..bb32782a9d3 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -117,7 +117,8 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
@@ -127,6 +128,9 @@ pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anyti
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 90237c70829..d26ff26e3e3 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -20,6 +20,7 @@
#include "utils/backend_status.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/pgstat_kind.h"
#include "utils/relcache.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h" /* for backward compatibility */ /* IWYU pragma: export */
@@ -536,10 +537,11 @@ extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
/*
- * Schedule the next anytime stats update timeout.
+ * Schedule the next anytime stats update timeout and mark that we have
+ * mixed anytime stats pending.
*
* This should be called whenever accumulating statistics that support
- * FLUSH_ANYTIME flushing mode.
+ * FLUSH_ANYTIME or FLUSH_MIXED flushing modes.
*/
#define pgstat_schedule_anytime_update() \
do { \
@@ -705,37 +707,58 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned += (n); \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_hit++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 607f4255268..1a2114aad8a 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -322,8 +322,10 @@ typedef struct PgStat_KindInfo
* that cannot use PgStat_EntryRef->pending.
*
* The anytime_only parameter indicates whether this is an anytime flush.
+ * The is_partial parameter indicates whether this is a partial flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait,
+ bool anytime_only, bool *is_partial);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -757,7 +759,8 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -765,7 +768,8 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -790,7 +794,8 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -858,7 +863,8 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..11e3e57806d 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..aef582e7582 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..47414eb6009 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -156,6 +162,8 @@ step s2_func_stats {
}
# Relation stats steps
+step s2_track_counts_on { SET track_counts = on; }
+step s2_track_counts_off { SET track_counts = off; }
step s2_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; }
step s2_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';}
@@ -435,6 +443,23 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_off
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_on
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index e9f1bda6b32..59f531df5f7 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -85,7 +85,8 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait, bool anytime_only);
+ bool nowait, bool anytime_only,
+ bool *is_partial);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -153,7 +154,8 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
@@ -161,6 +163,9 @@ test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
shared_entry = (PgStatShared_CustomVarEntry *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-23 02:12 ` Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-02-23 02:12 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> > I took a look at this today out of interest,
>
> Thanks!
>
> > so, instead of calling IS_INJECTION_POINT_ATTACHED macro which is
>
> I think that not calling IS_INJECTION_POINT_ATTACHED() but only relying on
> "ifdef USE_INJECTION_POINTS" would set the tiny timeout value to the entire test
> suit.
Yes, you are right about that.
To Michael's earlier comments:
> I don't find the design of this patch appealing, and my mind points
> towards two pieces of it:
> 1) The new requirement related to pgstat_schedule_anytime_update()
> that a stats kind needs to call to enable a timeout. This partially
> doubles with pgstat_report_fixed. And I suspect that this extra set
> of requirements, introducing a new level of complexity for in-core
> stats kinds as well as extension developers, would be the source of
> more bugs.
I don't see how for fixed stats, adding a
pgstat_schedule_anytime_update() call for such kinds
will be too complex or more error prone. it looks quite straightforward
as this is done before pgstat_report_fixed is set to true. Also,
because processes like wal-sender can loop forever, as mentioned
here [1], a timeout at stats_flush_interval seems like a
straightforward way to deal with this problem.
For variable-length statistics, perhaps we can do things a bit
differently than what is currently proposed. 0005 requires
a relation anytime stat update to call
pgstat_schedule_anytime_update(). This is done this way because
it allows long-running queries to update their stats every
stats_flush_interval using a timeout.
But maybe what we should be doing for variable-numbered stats is
to schedule an anytime update whenever a "transaction goes idle".
This way, unlike the current state of things where we are only
updating relation stats at the end of a transaction, we are now
updating relation stats at the end of SQL execution, and within a
transaction.
Basically, we will continue scheduling anytime
updates every stats_flush_interval for fixed stats (xlog,
bgwriter, etc.), but for variable stats, we only update anytime
stats after SQL execution. This is better than what we have now,
where stats are only updated at the end of the transaction.
The timeout will only be needed to schedule an update
for fixed stats. For variable stats, we can use
GetCurrentStatementStartTimestamp, which is the timestamp of the
last query executed, to throttle pgstat_flush_pending_entries().
We can also flush variable number stats whenever we flush
fixed number stats, in case we enter a long
idle-in-transaction state after a few quick back-to-back queries.
> 2) The timeout requirement itself, relying on a timeout threshold
> controlled by a backend-side configuration.
Perhaps we may not need the stats_flush_inteval and just force
a 10 second timeout for fixed stats.
> With that in mind, wouldn't it be simpler if we introduced an API that
> could be used from client applications instead, in a model similar
> what we do for procsignal.c/h? One such example is
> LOG_MEMORY_CONTEXT, where we have a SQL function that is able to tell
> to a backend that it needs to do something. I could see various
> benefits to this approach, because it gives more flexibility with the
> timing of the stats flushes, which may not be a backend-side only
> policy:
> - Use a cron bgworker in the backend, that scans pg_stat_activity, for
> example for long-running transactions based on a threshold.
> - Do the same periodic scan of pg_stat_activity, but from a client
> application.
I find it odd to ask applications/clients to trigger a flush. I am not saying
that we should not offer such an API, especially if someone want to flush
stats more frequently than stats_flush_interval, but there should be
the ability for core to handle this automatically outside of the transaction
boundaries.
One comment about the current test. I think there is a bug that was
missed in the earlier review. For the var_anytime_update, we need to
have an escape before the pipe. Also, we should set
stats_fetch_consistency=none in that test.
-like($result, qr/^entry2|2|/m,
+like($result, qr/^entry2\|2\|/m,
Otherwise, the test is returning a false-positive.
[1] https://www.postgresql.org/message-id/erpzwxoptqhuptdrtehqydzjapvroumkhh7lc6poclbhe7jk7l%40l3yfsq5q4...
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-23 08:14 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-23 08:14 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Sun, Feb 22, 2026 at 08:12:06PM -0600, Sami Imseih wrote:
> > > I took a look at this today out of interest,
> >
> > Thanks!
> >
> > > so, instead of calling IS_INJECTION_POINT_ATTACHED macro which is
> >
> > I think that not calling IS_INJECTION_POINT_ATTACHED() but only relying on
> > "ifdef USE_INJECTION_POINTS" would set the tiny timeout value to the entire test
> > suit.
>
> Yes, you are right about that.
And even if we are able to set a tiny timeout to say N in the test, the test
would still need a pg_sleep(N+<something>). So I wonder if the best way is not to
re-introduce pg_stat_force_anytime_flush() that was in v6?
> To Michael's earlier comments:
>
> For variable-length statistics, perhaps we can do things a bit
> differently than what is currently proposed. 0005 requires
> a relation anytime stat update to call
> pgstat_schedule_anytime_update(). This is done this way because
> it allows long-running queries to update their stats every
> stats_flush_interval using a timeout.
>
> But maybe what we should be doing for variable-numbered stats is
> to schedule an anytime update whenever a "transaction goes idle".
I think the logic for fixed stats and variable stats should be the same. If
not we could observe discrepancies: for example a long running select could
genereate reads/hits IO visible in pg_stat_io but tuples_returned, tuples_fetched,
blocks_fetched or blocks_hit would not be updated until the session goes idle.
> I find it odd to ask applications/clients to trigger a flush. I am not saying
> that we should not offer such an API, especially if someone want to flush
> stats more frequently than stats_flush_interval,
Yeah, and that could help with the tests to avoid the sleep.
> but there should be
> the ability for core to handle this automatically outside of the transaction
> boundaries.
Agreed. I think that public facing API could be an addition, not a replacement.
> One comment about the current test. I think there is a bug that was
> missed in the earlier review. For the var_anytime_update, we need to
> have an escape before the pipe.
>
> -like($result, qr/^entry2|2|/m,
> +like($result, qr/^entry2\|2\|/m,
Oh right, thanks! The attach rewrites it like the fixed test instead.
Also the attached introduces a mandatory rebase due to 308622edf17.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v11-0001-Add-pgstat_report_anytime_stat-for-periodic-stat.patch (42.9K, ../../aZwMdVBYobV41duU@ip-10-97-1-34.eu-west-3.compute.internal/2-v11-0001-Add-pgstat_report_anytime_stat-for-periodic-stat.patch)
download | inline diff:
From 88752c424a416952dc3ead0403eabfec5b19f0c6 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v11 1/5] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces
spikes when flushed.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducin
a new callback).
- Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush,
relying on PGSTAT_MIN_INTERVAL
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 6 +
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/replication/walsender.c | 8 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 121 +++++++++++++++---
src/backend/utils/activity/pgstat_backend.c | 13 +-
src/backend/utils/activity/pgstat_bgwriter.c | 2 +-
.../utils/activity/pgstat_checkpointer.c | 2 +-
src/backend/utils/activity/pgstat_database.c | 2 +-
src/backend/utils/activity/pgstat_function.c | 4 +-
src/backend/utils/activity/pgstat_io.c | 10 +-
src/backend/utils/activity/pgstat_relation.c | 12 +-
src/backend/utils/activity/pgstat_slru.c | 6 +-
.../utils/activity/pgstat_subscription.c | 4 +-
src/backend/utils/activity/pgstat_wal.c | 10 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 22 ++++
src/include/utils/pgstat_internal.h | 52 ++++++--
src/include/utils/timeout.h | 1 +
.../test_custom_stats/test_custom_var_stats.c | 4 +-
src/tools/pgindent/typedefs.list | 1 +
28 files changed, 279 insertions(+), 66 deletions(-)
10.5% src/backend/postmaster/
5.8% src/backend/replication/
51.0% src/backend/utils/activity/
5.8% src/backend/
18.7% src/include/utils/
6.6% src/include/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13cce9b49f1..cf29fc91f70 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0956bd39a85..059c601c3b8 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..e11c4b099c8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..4954fe425b7 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 742137edad6..f1bae9d23d6 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN); /* no query to cancel */
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 7c0e2809c17..bcf59227a00 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN); /* no query to cancel */
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 7c1b8757d7d..aecc7a127e6 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2cde8ebc729..a7214d0dc6f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
@@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d01a09dd0c4..8c30efa2443 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3564,6 +3564,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..ddd331e2c81 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -108,10 +108,12 @@
#include "pgstat.h"
#include "storage/fd.h"
#include "storage/ipc.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +124,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -218,6 +219,12 @@ PgStat_LocalState pgStatLocal;
*/
bool pgstat_report_fixed = false;
+/*
+ * Track when there is pending anytime flush to avoid relying on
+ * get_timeout_active() in hot pathes.
+ */
+bool pgstat_pending_anytime = false;
+
/* ----------
* Local data
*
@@ -288,6 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +330,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +346,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +364,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +382,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -436,6 +449,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +467,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +485,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +791,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,7 +1297,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
@@ -1345,9 +1350,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,8 +1387,22 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
@@ -1402,6 +1426,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait, anytime_only);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+
+ pgstat_pending_anytime = false;
+}
+
+/*
+ * Timeout handler for flushing anytime stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index f2f8d3ff75f..b09316d3ab3 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
* if some statistics could not be flushed due to lock contention.
*/
bool
-pgstat_flush_backend(bool nowait, bits32 flags)
+pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only)
{
PgStat_EntryRef *entry_ref;
bool has_pending_data = false;
@@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
* If some stats could not be flushed due to lock contention, return true.
*/
bool
-pgstat_backend_flush_cb(bool nowait)
+pgstat_backend_flush_cb(bool nowait, bool anytime_only)
{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
+ return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only);
}
/*
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index ed2fd801189..1c5f0c3ec40 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -61,7 +61,7 @@ pgstat_report_bgwriter(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 1f70194b7a7..2d89a082464 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -68,7 +68,7 @@ pgstat_report_checkpointer(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 933dcb5cae5..8e86df60461 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index e6b84283c6c..5ba4958382f 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
+ Assert(!anytime_only);
+
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..7cd32900236 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void)
* Simpler wrapper of pgstat_io_flush_cb()
*/
void
-pgstat_flush_io(bool nowait)
+pgstat_flush_io(bool nowait, bool anytime_only)
{
- (void) pgstat_io_flush_cb(nowait);
+ (void) pgstat_io_flush_cb(nowait, anytime_only);
}
/*
@@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait)
* acquired. Otherwise, return false.
*/
bool
-pgstat_io_flush_cb(bool nowait)
+pgstat_io_flush_cb(bool nowait, bool anytime_only)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..04d21483d93 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* is done -- which will likely vacuum many relations -- or until the
* VACUUM command has processed all tables and committed.
*/
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel,
pgstat_unlock_entry(entry_ref);
/* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
* entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
+ Assert(!anytime_only);
+
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..bf8a4d58673 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name)
* acquired. Otherwise return false.
*/
bool
-pgstat_slru_flush_cb(bool nowait)
+pgstat_slru_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
int i;
@@ -223,6 +224,9 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 3277cf88a4e..6b6eec7578d 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -117,11 +117,13 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
+ Assert(!anytime_only);
+
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..2c2f3f10e10 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -51,12 +51,12 @@ pgstat_report_wal(bool force)
nowait = !force;
/* flush wal stats */
- (void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
+ (void) pgstat_wal_flush_cb(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
/* flush IO stats */
- pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -88,7 +88,7 @@ pgstat_wal_have_pending(void)
* acquired. Otherwise return false.
*/
bool
-pgstat_wal_flush_cb(bool nowait)
+pgstat_wal_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
WalUsage wal_usage_diff = {0};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..eeeac1bf39a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..84e698da214 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 9bb777c3d5a..b011a315679 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,6 +34,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -532,8 +535,24 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
+/*
+ * Schedule the next anytime stats update timeout.
+ *
+ * This should be called whenever accumulating statistics that support
+ * FLUSH_ANYTIME flushing mode.
+ */
+#define pgstat_schedule_anytime_update() \
+ do { \
+ if (IsUnderPostmaster && !pgstat_pending_anytime) \
+ { \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ pgstat_pending_anytime = true; \
+ } \
+ } while (0)
+
extern void pgstat_reset_counters(void);
extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_reset_of_kind(PgStat_Kind kind);
@@ -806,6 +825,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
* Variables in pgstat.c
*/
+extern PGDLLIMPORT bool pgstat_pending_anytime;
+
/* GUC parameters */
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
@@ -849,4 +870,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime;
/* updated by the traffic cop and in errfinish() */
extern PGDLLIMPORT SessionEndType pgStatSessionEndCause;
+
#endif /* PGSTAT_H */
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..607f4255268 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * The mode of when to flush stats. See PgStat_FlushMode for more details.
+ *
+ * This member only has meaning for statistics kinds that accumulate
+ * pending stats and use flush callbacks. For kinds that write directly to
+ * shared memory (e.g., archiver, bgwriter, checkpointer), this member has
+ * no effect.
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo
*
* "pgstat_report_fixed" needs to be set to trigger the flush of pending
* stats.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_static_cb) (bool nowait);
+ bool (*flush_static_cb) (bool nowait, bool anytime_only);
/*
* For fixed-numbered statistics: Reset All.
@@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
@@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
-extern bool pgstat_flush_backend(bool nowait, bits32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
+extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only);
+extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_io.c
*/
-extern void pgstat_flush_io(bool nowait);
+extern void pgstat_flush_io(bool nowait, bool anytime_only);
-extern bool pgstat_io_flush_cb(bool nowait);
+extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
* Functions in pgstat_slru.c
*/
-extern bool pgstat_slru_flush_cb(bool nowait);
+extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void);
*/
extern void pgstat_wal_init_backend_cb(void);
-extern bool pgstat_wal_flush_cb(bool nowait);
+extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
@@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index da28afbd929..4c207611236 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -84,7 +84,7 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait);
+ bool nowait, bool anytime_only);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -151,7 +151,7 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1dbc4b96f51 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2271,6 +2271,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v11-0002-Add-anytime-flush-tests-for-custom-stats.patch (9.1K, ../../aZwMdVBYobV41duU@ip-10-97-1-34.eu-west-3.compute.internal/3-v11-0002-Add-anytime-flush-tests-for-custom-stats.patch)
download | inline diff:
From 7e6ab1d0ce578486fb96b55a56162f71fe5a370c Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 5 Feb 2026 05:54:34 +0000
Subject: [PATCH v11 2/5] Add anytime flush tests for custom stats
---
.../test_custom_stats/t/001_custom_stats.pl | 43 ++++++++++++++
.../test_custom_fixed_stats--1.0.sql | 5 ++
.../test_custom_fixed_stats.c | 57 +++++++++++++++++++
.../test_custom_var_stats--1.0.sql | 5 ++
.../test_custom_stats/test_custom_var_stats.c | 27 +++++++++
5 files changed, 137 insertions(+)
35.8% src/test/modules/test_custom_stats/t/
64.1% src/test/modules/test_custom_stats/
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 9e6a7a38577..6ba4022418f 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -156,5 +156,48 @@ $result = $node->safe_psql('postgres',
);
is($result, "0", "report of fixed-sized after manual reset");
+# Test FLUSH_ANYTIME mechanism with custom fixed stats
+# This verifies that custom stats can be flushed during a transaction
+
+# Reset stats first
+$node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+my $anytime_test = q[
+ BEGIN;
+ SET LOCAL stats_fetch_consistency = none;
+ -- Accumulate stats
+ select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select 'fixed_anytime:'||numcalls from test_custom_stats_fixed_report();
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^fixed_anytime:2/m,
+ "anytime fixed stats flushed during transaction");
+
+# Test FLUSH_ANYTIME mechanism with custom variable stats
+# This verifies that custom stats can be flushed during a transaction
+
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+$anytime_test = q[
+ BEGIN;
+ SET LOCAL stats_fetch_consistency = none;
+ -- Accumulate stats
+ select test_custom_stats_var_anytime_update('entry2');
+ select test_custom_stats_var_anytime_update('entry2');
+ -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ select pg_sleep(1.5);
+ -- Check
+ select 'var_anytime:'||calls from test_custom_stats_var_report('entry2');
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^var_anytime:2/m,
+ "anytime var stats flushed during transaction");
+
# Test completed successfully
done_testing();
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
index 69a93b5241f..da3a798f289 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
@@ -18,3 +18,8 @@ CREATE FUNCTION test_custom_stats_fixed_reset()
RETURNS void
AS 'MODULE_PATHNAME', 'test_custom_stats_fixed_reset'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_fixed_anytime_update()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
index 485e08e5c19..e7fbb2737ef 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
@@ -18,6 +18,7 @@
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
PG_MODULE_MAGIC_EXT(
@@ -44,11 +45,13 @@ typedef struct PgStatShared_CustomFixedEntry
static void test_custom_stats_fixed_init_shmem_cb(void *stats);
static void test_custom_stats_fixed_reset_all_cb(TimestampTz ts);
static void test_custom_stats_fixed_snapshot_cb(void);
+static bool test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only);
static const PgStat_KindInfo custom_stats = {
.name = "test_custom_fixed_stats",
.fixed_amount = true, /* exactly one entry */
.write_to_file = true, /* persist to stats file */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.shared_size = sizeof(PgStat_StatCustomFixedEntry),
.shared_data_off = offsetof(PgStatShared_CustomFixedEntry, stats),
@@ -57,8 +60,12 @@ static const PgStat_KindInfo custom_stats = {
.init_shmem_cb = test_custom_stats_fixed_init_shmem_cb,
.reset_all_cb = test_custom_stats_fixed_reset_all_cb,
.snapshot_cb = test_custom_stats_fixed_snapshot_cb,
+ .flush_static_cb = test_custom_stats_fixed_flush_cb,
};
+/* Pending statistics */
+static PgStat_StatCustomFixedEntry PendingCustomStats = {0};
+
/*
* Kind ID for test_custom_fixed_stats.
*/
@@ -142,6 +149,38 @@ test_custom_stats_fixed_snapshot_cb(void)
#undef FIXED_COMP
}
+/*
+ * test_custom_stats_fixed_flush_cb
+ * Flush pending stats to shared memory
+ */
+static bool
+test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only)
+{
+ PgStatShared_CustomFixedEntry *stats_shmem;
+
+ /* Nothing to flush if no calls were made */
+ if (PendingCustomStats.numcalls == 0)
+ return false;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_TEST_CUSTOM_FIXED_STATS);
+
+ if (!nowait)
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true;
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numcalls += PendingCustomStats.numcalls;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ LWLockRelease(&stats_shmem->lock);
+
+ /* Reset pending stats */
+ PendingCustomStats.numcalls = 0;
+
+ return false; /* successfully flushed */
+}
+
/*--------------------------------------------------------------------------
* SQL-callable functions
*--------------------------------------------------------------------------
@@ -223,3 +262,21 @@ test_custom_stats_fixed_report(PG_FUNCTION_ARGS)
/* Return as tuple */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/*
+ * test_custom_stats_fixed_anytime_update
+ * Increment call counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_fixed_anytime_update);
+Datum
+test_custom_stats_fixed_anytime_update(PG_FUNCTION_ARGS)
+{
+ /* Accumulate in pending stats */
+ PendingCustomStats.numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+ pgstat_report_fixed = true;
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
index 5ed8cfc2dcf..ed66d38981e 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
@@ -24,3 +24,8 @@ CREATE FUNCTION test_custom_stats_var_report(INOUT name TEXT,
RETURNS SETOF record
AS 'MODULE_PATHNAME', 'test_custom_stats_var_report'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_var_anytime_update(IN name TEXT)
+RETURNS void
+AS 'MODULE_PATHNAME', 'test_custom_stats_var_anytime_update'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 4c207611236..e9f1bda6b32 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -18,6 +18,7 @@
#include "storage/dsm_registry.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_var_stats",
@@ -108,6 +109,7 @@ static const PgStat_KindInfo custom_stats = {
.name = "test_custom_var_stats",
.fixed_amount = false, /* variable number of entries */
.write_to_file = true, /* persist across restarts */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.track_entry_count = true, /* count active entries */
.accessed_across_databases = true, /* global statistics */
.shared_size = sizeof(PgStatShared_CustomVarEntry),
@@ -690,3 +692,28 @@ test_custom_stats_var_report(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * test_custom_stats_var_anytime_update
+ * Increment custom statistic counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_var_anytime_update);
+Datum
+test_custom_stats_var_anytime_update(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_EntryRef *entry_ref;
+ PgStat_StatCustomVarEntry *pending_entry;
+
+ /* Get pending entry in local memory */
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, InvalidOid,
+ PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name), NULL);
+
+ pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
+ pending_entry->numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+
+ PG_RETURN_VOID();
+}
--
2.34.1
[text/x-diff] v11-0003-Add-GUC-to-specify-non-transactional-statistics-.patch (10.0K, ../../aZwMdVBYobV41duU@ip-10-97-1-34.eu-west-3.compute.internal/4-v11-0003-Add-GUC-to-specify-non-transactional-statistics-.patch)
download | inline diff:
From 205e22a432753760b3110e851955fd3421ff3564 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v11 3/5] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 32 +++++++++++++++++++
src/backend/utils/activity/pgstat.c | 13 ++++++++
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/backend/utils/misc/timeout.c | 6 ++++
src/include/pgstat.h | 6 ++--
src/include/utils/guc_hooks.h | 1 +
src/include/utils/timeout.h | 1 +
.../test_custom_stats/t/001_custom_stats.pl | 6 ++--
9 files changed, 70 insertions(+), 6 deletions(-)
51.0% doc/src/sgml/
10.6% src/backend/utils/activity/
15.9% src/backend/utils/misc/
3.6% src/include/utils/
9.0% src/include/
9.6% src/test/modules/test_custom_stats/t/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 20dbcaeb3ee..1eed71007a7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8929,6 +8929,38 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which certain statistics, which can be updated while a
+ transaction is in progress, are made visible. These include WAL activity
+ and I/O operations.
+ Such statistics are refreshed at the specified interval and can be observed
+ during active transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>
+ and
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is generally
+ the smallest practical value for long-running transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect statistics that are only reported at
+ transaction end, such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics are always
+ flushed at the end of a transaction.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ddd331e2c81..fd6ab0db16f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -124,6 +124,8 @@
* ----------
*/
+/* minimum interval non-forced stats flushes.*/
+#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -204,6 +206,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -2171,6 +2174,16 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_all_timeouts_initialized())
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 9507778415d..073e08c7892 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2801,6 +2801,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f938cc65a3a..8bd37a25b38 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -688,6 +688,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/backend/utils/misc/timeout.c b/src/backend/utils/misc/timeout.c
index ddba5dc607c..85c4260d1db 100644
--- a/src/backend/utils/misc/timeout.c
+++ b/src/backend/utils/misc/timeout.c
@@ -828,3 +828,9 @@ get_timeout_finish_time(TimeoutId id)
{
return all_timeouts[id].fin_time;
}
+
+bool
+get_all_timeouts_initialized(void)
+{
+ return all_timeouts_initialized;
+}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b011a315679..90237c70829 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,9 +34,6 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
-/* Minimum interval non-forced stats flushes */
-#define PGSTAT_MIN_INTERVAL 1000
-
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -548,7 +545,7 @@ extern void pgstat_force_next_flush(void);
do { \
if (IsUnderPostmaster && !pgstat_pending_anytime) \
{ \
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval); \
pgstat_pending_anytime = true; \
} \
} while (0)
@@ -831,6 +828,7 @@ extern PGDLLIMPORT bool pgstat_pending_anytime;
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 9c90670d9b8..9b5d2a90387 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 10723bb664c..fe7327de209 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -93,5 +93,6 @@ extern bool get_timeout_active(TimeoutId id);
extern bool get_timeout_indicator(TimeoutId id, bool reset_indicator);
extern TimestampTz get_timeout_start_time(TimeoutId id);
extern TimestampTz get_timeout_finish_time(TimeoutId id);
+extern bool get_all_timeouts_initialized(void);
#endif /* TIMEOUT_H */
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 6ba4022418f..920443487c0 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -164,11 +164,12 @@ $node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
my $anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
SET LOCAL stats_fetch_consistency = none;
-- Accumulate stats
select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select 'fixed_anytime:'||numcalls from test_custom_stats_fixed_report();
@@ -184,12 +185,13 @@ like($result, qr/^fixed_anytime:2/m,
$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
$anytime_test = q[
+ SET stats_flush_interval = '1s';
BEGIN;
SET LOCAL stats_fetch_consistency = none;
-- Accumulate stats
select test_custom_stats_var_anytime_update('entry2');
select test_custom_stats_var_anytime_update('entry2');
- -- Wait (has to be greater than PGSTAT_MIN_INTERVAL)
+ -- Wait (has to be greater than stats_flush_interval)
select pg_sleep(1.5);
-- Check
select 'var_anytime:'||calls from test_custom_stats_var_report('entry2');
--
2.34.1
[text/x-diff] v11-0004-Remove-useless-calls-to-flush-some-stats.patch (7.7K, ../../aZwMdVBYobV41duU@ip-10-97-1-34.eu-west-3.compute.internal/5-v11-0004-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From aeac6f996ef494124c5888ce2699e9d7b03c435a Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v11 4/5] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index aecc7a127e6..edf5ac65660 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..cfa095ff0a8 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..29bae5e1121 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval = '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v11-0005-Change-RELATION-and-DATABASE-stats-to-anytime-fl.patch (34.2K, ../../aZwMdVBYobV41duU@ip-10-97-1-34.eu-west-3.compute.internal/6-v11-0005-Change-RELATION-and-DATABASE-stats-to-anytime-fl.patch)
download | inline diff:
From 543cade27fcc4913f53fbc8df5df87936f7d8294 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v11 5/5] Change RELATION and DATABASE stats to anytime flush
This commit allows mixing fields with different transaction behavior within
the same RELATION or DATABASE statistics kind: some fields are transactional
(e.g., tuple inserts/updates/deletes) while others are non-transactional
(e.g., sequential scans, blocks read).
It modifies the relation flush callback to handle the anytime_only parameter
introduced in commit <nnnn>.
Implementation details:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Change DATABASE from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Add a is_partial parameter to flush_pending_cb() to be able to distinguish
partial flushes in pgstat_flush_pending_entries()
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats. When set to true, it clears flushed fields from pending stats to
prevent double-counting at transaction boundary
DATABASE stats inherit the anytime flush behavior so that relation-derived
stats (tuples_returned, tuples_fetched, blocks_fetched, blocks_hit) are
visible while transactions are in progress.
Tests are added to verify the anytime flush behavior for mixed fields.
---
doc/src/sgml/monitoring.sgml | 37 ++++++-
src/backend/utils/activity/pgstat.c | 15 +--
src/backend/utils/activity/pgstat_database.c | 6 +-
src/backend/utils/activity/pgstat_function.c | 6 +-
src/backend/utils/activity/pgstat_relation.c | 92 ++++++++++++----
.../utils/activity/pgstat_subscription.c | 6 +-
src/include/pgstat.h | 27 ++++-
src/include/utils/pgstat_internal.h | 16 ++-
src/test/isolation/expected/stats.out | 102 ++++++++++++++++++
src/test/isolation/expected/stats_1.out | 102 ++++++++++++++++++
src/test/isolation/specs/stats.spec | 27 ++++-
.../test_custom_stats/test_custom_var_stats.c | 9 +-
12 files changed, 404 insertions(+), 41 deletions(-)
11.7% doc/src/sgml/
26.8% src/backend/utils/activity/
4.2% src/include/utils/
5.4% src/include/
45.1% src/test/isolation/expected/
4.7% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..f2321b631b0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -3956,8 +3969,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_seq_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last sequential scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last sequential scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -3984,8 +3997,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_idx_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last index scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last index scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index fd6ab0db16f..a8a905640d0 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -298,7 +298,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -316,7 +316,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -1354,7 +1354,8 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries. For entries
+ * that support it, the callback may flush only non-transactional fields.
* This is safe to call inside transactions.
*
* If anytime_only is false, flushes all entries.
@@ -1385,6 +1386,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1405,7 +1407,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
}
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait,
+ anytime_only, &is_partial_flush);
Assert(did_flush || nowait);
@@ -1415,8 +1418,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 8e86df60461..59dd0790fd7 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,8 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
@@ -443,6 +444,9 @@ pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
pendingent = (PgStat_StatDBEntry *) entry_ref->pending;
sharedent = (PgStatShared_Database *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index 5ba4958382f..44193c93fc7 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,7 +190,8 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
@@ -200,6 +201,9 @@ pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ae2952cae89..62363dacfe1 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -47,7 +47,19 @@ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_lev
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
+static void flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry,
+ PgStat_TableCounts *counts, bool anytime_only);
+/*
+ * Update database statistics with non-transactional stats.
+ */
+#define UPDATE_DATABASE_ANYTIME_STATS(dbentry, counts) \
+ do { \
+ (dbentry)->tuples_returned += (counts)->tuples_returned; \
+ (dbentry)->tuples_fetched += (counts)->tuples_fetched; \
+ (dbentry)->blocks_fetched += (counts)->blocks_fetched; \
+ (dbentry)->blocks_hit += (counts)->blocks_hit; \
+ } while (0)
/*
* Copy stats between relations. This is used for things like REINDEX
@@ -789,6 +801,29 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Helper function to flush non-transactional statistics.
+ */
+static void
+flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry, PgStat_TableCounts *counts,
+ bool anytime_only)
+{
+ TimestampTz t;
+
+ tabentry->numscans += counts->numscans;
+ if (counts->numscans)
+ {
+ t = anytime_only ? GetCurrentTimestamp() : GetCurrentTransactionStopTimestamp();
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+
+ tabentry->tuples_returned += counts->tuples_returned;
+ tabentry->tuples_fetched += counts->tuples_fetched;
+ tabentry->blocks_fetched += counts->blocks_fetched;
+ tabentry->blocks_hit += counts->blocks_hit;
+}
+
/*
* Flush out pending stats for the entry
*
@@ -797,9 +832,17 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
*
* Some of the stats are copied to the corresponding pending database stats
* entry when successfully flushing.
+ *
+ * If anytime_only is true, only non-transactional fields are flushed
+ * (numscans, tuples_returned, tuples_fetched, blocks_fetched, blocks_hit).
+ * Transactional fields remain pending until transaction boundary.
+ *
+ * Some of the stats are copied to the corresponding pending database stats
+ * entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -807,12 +850,13 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
- Assert(!anytime_only);
-
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+ /* this is a partial flush if in anytime only mode */
+ *is_partial = anytime_only;
+
/*
* Ignore entries that didn't accumulate any actual counts, such as
* indexes that were opened by the planner but not used.
@@ -824,19 +868,36 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
- /* add the values to the shared entry. */
tabentry = &shtabstats->stats;
- tabentry->numscans += lstats->counts.numscans;
- if (lstats->counts.numscans)
+ if (anytime_only)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
- if (t > tabentry->lastscan)
- tabentry->lastscan = t;
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, true);
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
+ /*
+ * Clear the flushed fields from pending stats to prevent
+ * double-counting when we flush all fields at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
}
- tabentry->tuples_returned += lstats->counts.tuples_returned;
- tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, false);
+
tabentry->tuples_inserted += lstats->counts.tuples_inserted;
tabentry->tuples_updated += lstats->counts.tuples_updated;
tabentry->tuples_deleted += lstats->counts.tuples_deleted;
@@ -866,9 +927,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
*/
tabentry->ins_since_vacuum += lstats->counts.tuples_inserted;
- tabentry->blocks_fetched += lstats->counts.blocks_fetched;
- tabentry->blocks_hit += lstats->counts.blocks_hit;
-
/* Clamp live_tuples in case of negative delta_live_tuples */
tabentry->live_tuples = Max(tabentry->live_tuples, 0);
/* Likewise for dead_tuples */
@@ -878,13 +936,11 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
/* The entry was successfully flushed, add the same to database stats */
dbentry = pgstat_prep_database_pending(dboid);
- dbentry->tuples_returned += lstats->counts.tuples_returned;
- dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
dbentry->tuples_inserted += lstats->counts.tuples_inserted;
dbentry->tuples_updated += lstats->counts.tuples_updated;
dbentry->tuples_deleted += lstats->counts.tuples_deleted;
- dbentry->blocks_fetched += lstats->counts.blocks_fetched;
- dbentry->blocks_hit += lstats->counts.blocks_hit;
return true;
}
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 6b6eec7578d..bb32782a9d3 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -117,7 +117,8 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
@@ -127,6 +128,9 @@ pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anyti
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 90237c70829..d26ff26e3e3 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -20,6 +20,7 @@
#include "utils/backend_status.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/pgstat_kind.h"
#include "utils/relcache.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h" /* for backward compatibility */ /* IWYU pragma: export */
@@ -536,10 +537,11 @@ extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
/*
- * Schedule the next anytime stats update timeout.
+ * Schedule the next anytime stats update timeout and mark that we have
+ * mixed anytime stats pending.
*
* This should be called whenever accumulating statistics that support
- * FLUSH_ANYTIME flushing mode.
+ * FLUSH_ANYTIME or FLUSH_MIXED flushing modes.
*/
#define pgstat_schedule_anytime_update() \
do { \
@@ -705,37 +707,58 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned += (n); \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_hit++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 607f4255268..1a2114aad8a 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -322,8 +322,10 @@ typedef struct PgStat_KindInfo
* that cannot use PgStat_EntryRef->pending.
*
* The anytime_only parameter indicates whether this is an anytime flush.
+ * The is_partial parameter indicates whether this is a partial flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait,
+ bool anytime_only, bool *is_partial);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -757,7 +759,8 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -765,7 +768,8 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -790,7 +794,8 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -858,7 +863,8 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..11e3e57806d 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..aef582e7582 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..47414eb6009 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -156,6 +162,8 @@ step s2_func_stats {
}
# Relation stats steps
+step s2_track_counts_on { SET track_counts = on; }
+step s2_track_counts_off { SET track_counts = off; }
step s2_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; }
step s2_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';}
@@ -435,6 +443,23 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_off
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_on
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index e9f1bda6b32..59f531df5f7 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -85,7 +85,8 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait, bool anytime_only);
+ bool nowait, bool anytime_only,
+ bool *is_partial);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -153,7 +154,8 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool anytime_only, bool *is_partial)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
@@ -161,6 +163,9 @@ test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
shared_entry = (PgStatShared_CustomVarEntry *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
--
2.34.1
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-23 23:47 ` Sami Imseih <samimseih@gmail.com>
2026-02-24 01:56 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-24 12:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 2 replies; 71+ messages in thread
From: Sami Imseih @ 2026-02-23 23:47 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> > For variable-length statistics, perhaps we can do things a bit
> > differently than what is currently proposed. 0005 requires
> > a relation anytime stat update to call
> > pgstat_schedule_anytime_update(). This is done this way because
> > it allows long-running queries to update their stats every
> > stats_flush_interval using a timeout.
> >
> > But maybe what we should be doing for variable-numbered stats is
> > to schedule an anytime update whenever a "transaction goes idle".
>
> I think the logic for fixed stats and variable stats should be the same. If
> not we could observe discrepancies: for example a long running select could
> genereate reads/hits IO visible in pg_stat_io but tuples_returned, tuples_fetched,
> blocks_fetched or blocks_hit would not be updated until the session goes idle.
After having more time to think about this, I believe it can be much simpler.
As soon as we enter an idle-in-transaction (aborted) state, we can simply
schedule an anytime update. This ensures that a flush is scheduled whenever
the fixed stats trigger one, which will likely be the most common reason
(e.g., I/O stats, WAL stats, etc.). To cover the cases where fixed stats
do not schedule a flush, we can also schedule one as soon as a transaction
goes idle.
In my mind, this makes this whole flushing scheduling behavior easy to reason
about, and if we introduce future anytime stats anywhere, we are not required
to schedule a flush for each individual field. The flush callback will of course
still need to decide what to flush anytime or at the transaction boundary.
What do you think?
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-24 01:56 ` Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 71+ messages in thread
From: Michael Paquier @ 2026-02-24 01:56 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Mon, Feb 23, 2026 at 05:47:22PM -0600, Sami Imseih wrote:
>> I think the logic for fixed stats and variable stats should be the same. If
>> not we could observe discrepancies: for example a long running select could
>> genereate reads/hits IO visible in pg_stat_io but tuples_returned, tuples_fetched,
>> blocks_fetched or blocks_hit would not be updated until the session goes idle.
>
> After having more time to think about this, I believe it can be much simpler.
> As soon as we enter an idle-in-transaction (aborted) state, we can simply
> schedule an anytime update. This ensures that a flush is scheduled whenever
> the fixed stats trigger one, which will likely be the most common reason
> (e.g., I/O stats, WAL stats, etc.). To cover the cases where fixed stats
> do not schedule a flush, we can also schedule one as soon as a transaction
> goes idle.
>
> In my mind, this makes this whole flushing scheduling behavior easy to reason
> about, and if we introduce future anytime stats anywhere, we are not required
> to schedule a flush for each individual field. The flush callback will of course
> still need to decide what to flush anytime or at the transaction boundary.
>
> What do you think?
I cannot picture yet fully how a patch among these lines would be
shaped, but having a strategic flush of the stats when we are in an
idle-in-transaction state sounds like an interesting option here.
I think that this leans towards two first pieces of infrastructure for
this patch set:
- The new stats kind option.
- A new pgstats API that is able to classify the flushes depending on
property assigned for each stats kind, and make these happen on a
caller-basis.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aZ0FQNIP8-bIA2FI@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-02-24 12:01 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 06:26 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-24 12:01 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Mon, Feb 23, 2026 at 05:47:22PM -0600, Sami Imseih wrote:
> > > For variable-length statistics, perhaps we can do things a bit
> > > differently than what is currently proposed. 0005 requires
> > > a relation anytime stat update to call
> > > pgstat_schedule_anytime_update(). This is done this way because
> > > it allows long-running queries to update their stats every
> > > stats_flush_interval using a timeout.
> > >
> > > But maybe what we should be doing for variable-numbered stats is
> > > to schedule an anytime update whenever a "transaction goes idle".
> >
> > I think the logic for fixed stats and variable stats should be the same. If
> > not we could observe discrepancies: for example a long running select could
> > genereate reads/hits IO visible in pg_stat_io but tuples_returned, tuples_fetched,
> > blocks_fetched or blocks_hit would not be updated until the session goes idle.
>
> After having more time to think about this, I believe it can be much simpler.
> As soon as we enter an idle-in-transaction (aborted) state, we can simply
> schedule an anytime update. This ensures that a flush is scheduled whenever
> the fixed stats trigger one, which will likely be the most common reason
> (e.g., I/O stats, WAL stats, etc.). To cover the cases where fixed stats
> do not schedule a flush, we can also schedule one as soon as a transaction
> goes idle.
>
> In my mind, this makes this whole flushing scheduling behavior easy to reason
> about, and if we introduce future anytime stats anywhere, we are not required
> to schedule a flush for each individual field. The flush callback will of course
> still need to decide what to flush anytime or at the transaction boundary.
>
> What do you think?
My understanding is that (correct me if I'm wrong):
- fixed stats would still be designed the way it is in v11
- variable stats would not need the pgstat_schedule_anytime_update() calls in
various places. The flush would be done/schedule when the session goes idle.
Then I agree that that looks ok and that:
> This ensures that a flush is scheduled whenever
> the fixed stats trigger one, which will likely be the most common reason
> (e.g., I/O stats, WAL stats, etc.)
Though I don't think that adresses Michael's concern: "main worries are
mainly around 1), I guess, with the new SIGALRM handler requirements for all
auxiliary processes" in [1].
Regards,
[1]: https://postgr.es/m/aZznT84Ssh8PywcH%40paquier.xyz
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-24 12:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-03-16 06:26 ` Michael Paquier <michael@paquier.xyz>
2026-03-16 09:20 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Michael Paquier @ 2026-03-16 06:26 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Tue, Feb 24, 2026 at 12:01:30PM +0000, Bertrand Drouvot wrote:
> Though I don't think that adresses Michael's concern: "main worries are
> mainly around 1), I guess, with the new SIGALRM handler requirements for all
> auxiliary processes" in [1].
FWIW, I am still concerned about that, and I have pondered about what
we could do here. While reviewing the existing code, one thing that I
have noticed we could do is rely on the existing interface of
pgstat_report_stat() without changing the existing callers, and not
touching at all the flush callbacks. If we begin to require the
"force" mode when the routine the called inside a transaction block,
things seem to work pretty smoothly in combination with a stats kind
property that allows the stats data to be flushed if we are inside a
transaction while a report happens. And it is possible to enforce
checks inside pgstat_report_stat() as well.
So please find attached my shot at that:
- Introduction of a new system function called pg_stat_report(), based
on a procsignal that gives a way to signal backends for a stats
update, reusing the existing code where we only do flushes when idle
and not in a transaction.
- Property that tracks under which contexts the reports are allowed.
Here I have decided to stick with simple, as in only allowing IO and
WAL stats to be flushed if we are inside a transaction.
Using that, I have done a few tests with three backends:
- One with a long-running transaction.
- One that periodically triggers the reports.
- One that looks at IO and WAL stat.
And the third session is able to get refreshes for both of these stats
kinds, while the other stats remain the same.
Note that this is a WIP, which is check-world stable. One thing that
sticks a bit in mind now is that perhaps we should not allow the
function for auxiliary processes at all. A second thing is the
requirement of allowing partial flushes at the end of the report path,
which is OK because the variable-sized stats can have pending data.
Perhaps we should just have pgstat_flush_pending_entries() provide a
correct status in line with the property set in a stats kind when we
try a flush while in a transaction.
Thoughts or tomatoes?
--
Michael
From c5aeb083265efbd6041ed3868b669997d4430760 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 16 Mar 2026 15:12:24 +0900
Subject: [PATCH] Add support for transient stats updates
This introduces a function able to push stats updates, with a new stats
kind property to allow stats to be updated while in a transaction.
---
src/include/catalog/pg_proc.dat | 7 +++
src/include/miscadmin.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/pgstat_internal.h | 16 ++++++
src/backend/storage/ipc/procsignal.c | 16 ++++++
src/backend/tcop/postgres.c | 11 +++-
src/backend/utils/activity/pgstat.c | 42 ++++++++++++--
src/backend/utils/adt/pgstatfuncs.c | 58 ++++++++++++++++++++
src/backend/utils/init/globals.c | 1 +
src/test/regress/expected/misc_functions.out | 52 ++++++++++++++++++
src/test/regress/sql/misc_functions.sql | 29 ++++++++++
doc/src/sgml/func/func-admin.sgml | 16 ++++++
12 files changed, 242 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 361e2cfffebe..85869154657a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8692,6 +8692,13 @@
prosrc => 'pg_log_backend_memory_contexts',
proacl => '{POSTGRES=X}' },
+# request an update of statistics
+{ oid => '8789', descr => 'have the specified backend push a pgstats update',
+ proname => 'pg_stat_report', provolatile => 'v',
+ prorettype => 'bool', proargtypes => 'int4',
+ prosrc => 'pg_stat_report',
+ proacl => '{POSTGRES=X}' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9b..a245cdd79d17 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -94,6 +94,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t TransactionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalStatsUpdatePending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 348fba53a931..d2e60403ad52 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_STATS_UPDATE, /* pgstats update */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
PROCSIG_RECOVERY_CONFLICT, /* backend is blocking recovery, check
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed5..fa4cc4fe9c60 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,17 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Contexts related to the report of the statistics, defined as properties
+ * of PgStat_KindInfo.report_context. These define when a stats report is
+ * allowed depending on the stats kind and the context where
+ * pgstat_report_stat() is called.
+ */
+
+/* report allowed while idle, outside a transaction (default) */
+#define PGSTAT_REPORT_IDLE 0x00
+/* report of stats data allowed within a transaction */
+#define PGSTAT_REPORT_TRANSACTION 0x01
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +262,11 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * Contexts allowed for the report of this stats kind data.
+ */
+ bits32 report_context;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 7e017c8d53b5..e1bff2185933 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -490,6 +490,19 @@ HandleProcSignalBarrierInterrupt(void)
/* latch will be set by procsignal_sigusr1_handler */
}
+/*
+ * Handle receipt of an interrupt indicating that a stats update has been
+ * requested. This routine only gets called when PROCSIG_STATS_UPDATE is
+ * sent.
+ */
+static void
+HandleProcSignalStatsUpdateInterrupt(void)
+{
+ InterruptPending = true;
+ ProcSignalStatsUpdatePending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
/*
* Perform global barrier related interrupt checking.
*
@@ -694,6 +707,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_STATS_UPDATE))
+ HandleProcSignalStatsUpdateInterrupt();
+
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d01a09dd0c41..ddd57dfea780 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3555,12 +3555,17 @@ ProcessInterrupts(void)
/*
* If there are pending stats updates and we currently are truly idle
- * (matching the conditions in PostgresMain(), report stats now.
+ * (matching the conditions in PostgresMain(), or if a status update has
+ * been requested, report stats now.
*/
if (IdleStatsUpdateTimeoutPending &&
- DoingCommandRead && !IsTransactionOrTransactionBlock())
+ DoingCommandRead && !IsTransactionOrTransactionBlock() ||
+ ProcSignalStatsUpdatePending)
{
- IdleStatsUpdateTimeoutPending = false;
+ if (IdleStatsUpdateTimeoutPending)
+ IdleStatsUpdateTimeoutPending = false;
+ if (ProcSignalStatsUpdatePending)
+ ProcSignalStatsUpdatePending = false;
pgstat_report_stat(true);
}
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5ad..5521e96d0cae 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -436,6 +436,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .report_context = PGSTAT_REPORT_TRANSACTION,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -470,6 +471,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .report_context = PGSTAT_REPORT_TRANSACTION,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -698,8 +700,9 @@ pgstat_initialize(void)
* a timeout after which to call pgstat_report_stat(true), but are not
* required to do so.
*
- * Note that this is called only when not within a transaction, so it is fair
- * to use transaction stop time as an approximation of current time.
+ * Note that when this is called only when not within a transaction, we use
+ * the transaction stop time as an approximation of current time. "force"
+ * is required when this is called within a transaction.
*/
long
pgstat_report_stat(bool force)
@@ -709,9 +712,14 @@ pgstat_report_stat(bool force)
bool partial_flush;
TimestampTz now;
bool nowait;
+ bool is_xact = IsTransactionOrTransactionBlock();
pgstat_assert_is_up();
- Assert(!IsTransactionOrTransactionBlock());
+ /*
+ * "force" is required if this routine is called inside a transaction
+ * block.
+ */
+ Assert(!is_xact || force);
/* "absorb" the forced flush even if there's nothing to flush */
if (pgStatForceNextFlush)
@@ -789,6 +797,11 @@ pgstat_report_stat(bool force)
if (!kind_info->flush_static_cb)
continue;
+ /* Skip if this stats kind cannot be flushed in a transaction */
+ if (is_xact &&
+ (kind_info->report_context & PGSTAT_REPORT_TRANSACTION) == 0)
+ continue;
+
partial_flush |= kind_info->flush_static_cb(nowait);
}
}
@@ -801,8 +814,11 @@ pgstat_report_stat(bool force)
*/
if (partial_flush)
{
- /* force should have prevented us from getting here */
- Assert(!force);
+ /*
+ * force should have prevented us from getting here, and partial
+ * flushes are accepted inside a transaction.
+ */
+ Assert(!force || is_xact);
/* remember since when stats have been pending */
if (pending_since == 0)
@@ -1351,6 +1367,7 @@ pgstat_flush_pending_entries(bool nowait)
{
bool have_pending = false;
dlist_node *cur = NULL;
+ bool is_xact = IsTransactionOrTransactionBlock();
/*
* Need to be a bit careful iterating over the list of pending entries.
@@ -1377,6 +1394,21 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip if this stats kind cannot be flushed while in a transaction */
+ if (is_xact &&
+ (kind_info->report_context & PGSTAT_REPORT_TRANSACTION) == 0)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index bad5642d9c90..b27a5eb58e18 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -28,6 +28,7 @@
#include "replication/logicallauncher.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "storage/procsignal.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/timestamp.h"
@@ -2325,3 +2326,60 @@ pg_stat_have_stats(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(pgstat_have_entry(kind, dboid, objid));
}
+
+/*
+ * pg_stat_report
+ * Signal a backend or an auxiliary process to have it push an update
+ * of its statistics data.
+ *
+ * By default, only superusers are allowed to signal to log the memory
+ * contexts because allowing any users to issue this request at an unbounded
+ * rate would cause lots of log messages and which can lead to denial of
+ * service. Additional roles can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to update their statistics.
+ */
+Datum
+pg_stat_report(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ proc = AuxiliaryPidGetProc(pid);
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; but by the time we reach kill(), a process for which we
+ * get a valid proc here might have terminated on its own. This is OK,
+ * as at shutdown processes flush their stats.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+ if (SendProcSignal(pid, PROCSIG_STATS_UPDATE, procNumber) < 0)
+ {
+ /* Again, just a warning to allow loops */
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ PG_RETURN_BOOL(true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b3602..22960ee3b27b 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -38,6 +38,7 @@ volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalStatsUpdatePending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 6c03b1a79d75..68a0e1e02bcc 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -397,6 +397,58 @@ REVOKE EXECUTE ON FUNCTION pg_log_backend_memory_contexts(integer)
FROM regress_log_memory;
DROP ROLE regress_log_memory;
--
+-- pg_stat_report
+--
+-- check execution
+SELECT pg_stat_report(pg_backend_pid());
+ pg_stat_report
+----------------
+ t
+(1 row)
+
+SELECT pg_stat_report(pid) FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer';
+ pg_stat_report
+----------------
+ t
+(1 row)
+
+-- Check privileges
+CREATE ROLE regress_stat_report;
+SELECT has_function_privilege('regress_stat_report',
+ 'pg_stat_report(integer)', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+-- Fails
+SET ROLE regress_stat_report;
+SELECT pg_stat_report(pg_backend_pid());
+ERROR: permission denied for function pg_stat_report
+RESET ROLE;
+-- Access granted, then function works
+GRANT EXECUTE ON FUNCTION pg_stat_report(integer)
+ TO regress_stat_report;
+SELECT has_function_privilege('regress_stat_report',
+ 'pg_stat_report(integer)', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SET ROLE regress_stat_report;
+SELECT pg_stat_report(pg_backend_pid());
+ pg_stat_report
+----------------
+ t
+(1 row)
+
+RESET ROLE;
+REVOKE EXECUTE ON FUNCTION pg_stat_report(integer)
+ FROM regress_stat_report;
+DROP ROLE regress_stat_report;
+--
-- Test some built-in SRFs
--
-- The outputs of these are variable, so we can't just print their results
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 35b7983996c4..fe366904c9e8 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -154,6 +154,35 @@ REVOKE EXECUTE ON FUNCTION pg_log_backend_memory_contexts(integer)
DROP ROLE regress_log_memory;
+--
+-- pg_stat_report
+--
+
+-- check execution
+SELECT pg_stat_report(pg_backend_pid());
+SELECT pg_stat_report(pid) FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer';
+
+-- Check privileges
+CREATE ROLE regress_stat_report;
+SELECT has_function_privilege('regress_stat_report',
+ 'pg_stat_report(integer)', 'EXECUTE'); -- no
+-- Fails
+SET ROLE regress_stat_report;
+SELECT pg_stat_report(pg_backend_pid());
+RESET ROLE;
+-- Access granted, then function works
+GRANT EXECUTE ON FUNCTION pg_stat_report(integer)
+ TO regress_stat_report;
+SELECT has_function_privilege('regress_stat_report',
+ 'pg_stat_report(integer)', 'EXECUTE'); -- yes
+SET ROLE regress_stat_report;
+SELECT pg_stat_report(pg_backend_pid());
+RESET ROLE;
+REVOKE EXECUTE ON FUNCTION pg_stat_report(integer)
+ FROM regress_stat_report;
+DROP ROLE regress_stat_report;
+
--
-- Test some built-in SRFs
--
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 210b1118bdf7..114202c4fc19 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -220,6 +220,22 @@
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_stat_report</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Requests to update the statistics computed by the backend with the
+ specified process ID. This function can send the request to
+ backends and auxiliary processes except logger, pushing an
+ update of the statistics data.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
--
2.53.0
Attachments:
[text/plain] 0001-Add-support-for-transient-stats-updates.patch (16.6K, ../../abeimWKYq-n34FrH@paquier.xyz/2-0001-Add-support-for-transient-stats-updates.patch)
download | inline diff:
From c5aeb083265efbd6041ed3868b669997d4430760 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 16 Mar 2026 15:12:24 +0900
Subject: [PATCH] Add support for transient stats updates
This introduces a function able to push stats updates, with a new stats
kind property to allow stats to be updated while in a transaction.
---
src/include/catalog/pg_proc.dat | 7 +++
src/include/miscadmin.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/pgstat_internal.h | 16 ++++++
src/backend/storage/ipc/procsignal.c | 16 ++++++
src/backend/tcop/postgres.c | 11 +++-
src/backend/utils/activity/pgstat.c | 42 ++++++++++++--
src/backend/utils/adt/pgstatfuncs.c | 58 ++++++++++++++++++++
src/backend/utils/init/globals.c | 1 +
src/test/regress/expected/misc_functions.out | 52 ++++++++++++++++++
src/test/regress/sql/misc_functions.sql | 29 ++++++++++
doc/src/sgml/func/func-admin.sgml | 16 ++++++
12 files changed, 242 insertions(+), 8 deletions(-)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 361e2cfffebe..85869154657a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8692,6 +8692,13 @@
prosrc => 'pg_log_backend_memory_contexts',
proacl => '{POSTGRES=X}' },
+# request an update of statistics
+{ oid => '8789', descr => 'have the specified backend push a pgstats update',
+ proname => 'pg_stat_report', provolatile => 'v',
+ prorettype => 'bool', proargtypes => 'int4',
+ prosrc => 'pg_stat_report',
+ proacl => '{POSTGRES=X}' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9b..a245cdd79d17 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -94,6 +94,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t TransactionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalStatsUpdatePending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 348fba53a931..d2e60403ad52 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_STATS_UPDATE, /* pgstats update */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
PROCSIG_RECOVERY_CONFLICT, /* backend is blocking recovery, check
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed5..fa4cc4fe9c60 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,17 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Contexts related to the report of the statistics, defined as properties
+ * of PgStat_KindInfo.report_context. These define when a stats report is
+ * allowed depending on the stats kind and the context where
+ * pgstat_report_stat() is called.
+ */
+
+/* report allowed while idle, outside a transaction (default) */
+#define PGSTAT_REPORT_IDLE 0x00
+/* report of stats data allowed within a transaction */
+#define PGSTAT_REPORT_TRANSACTION 0x01
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +262,11 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * Contexts allowed for the report of this stats kind data.
+ */
+ bits32 report_context;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 7e017c8d53b5..e1bff2185933 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -490,6 +490,19 @@ HandleProcSignalBarrierInterrupt(void)
/* latch will be set by procsignal_sigusr1_handler */
}
+/*
+ * Handle receipt of an interrupt indicating that a stats update has been
+ * requested. This routine only gets called when PROCSIG_STATS_UPDATE is
+ * sent.
+ */
+static void
+HandleProcSignalStatsUpdateInterrupt(void)
+{
+ InterruptPending = true;
+ ProcSignalStatsUpdatePending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
/*
* Perform global barrier related interrupt checking.
*
@@ -694,6 +707,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_STATS_UPDATE))
+ HandleProcSignalStatsUpdateInterrupt();
+
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d01a09dd0c41..ddd57dfea780 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3555,12 +3555,17 @@ ProcessInterrupts(void)
/*
* If there are pending stats updates and we currently are truly idle
- * (matching the conditions in PostgresMain(), report stats now.
+ * (matching the conditions in PostgresMain(), or if a status update has
+ * been requested, report stats now.
*/
if (IdleStatsUpdateTimeoutPending &&
- DoingCommandRead && !IsTransactionOrTransactionBlock())
+ DoingCommandRead && !IsTransactionOrTransactionBlock() ||
+ ProcSignalStatsUpdatePending)
{
- IdleStatsUpdateTimeoutPending = false;
+ if (IdleStatsUpdateTimeoutPending)
+ IdleStatsUpdateTimeoutPending = false;
+ if (ProcSignalStatsUpdatePending)
+ ProcSignalStatsUpdatePending = false;
pgstat_report_stat(true);
}
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5ad..5521e96d0cae 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -436,6 +436,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .report_context = PGSTAT_REPORT_TRANSACTION,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -470,6 +471,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .report_context = PGSTAT_REPORT_TRANSACTION,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -698,8 +700,9 @@ pgstat_initialize(void)
* a timeout after which to call pgstat_report_stat(true), but are not
* required to do so.
*
- * Note that this is called only when not within a transaction, so it is fair
- * to use transaction stop time as an approximation of current time.
+ * Note that when this is called only when not within a transaction, we use
+ * the transaction stop time as an approximation of current time. "force"
+ * is required when this is called within a transaction.
*/
long
pgstat_report_stat(bool force)
@@ -709,9 +712,14 @@ pgstat_report_stat(bool force)
bool partial_flush;
TimestampTz now;
bool nowait;
+ bool is_xact = IsTransactionOrTransactionBlock();
pgstat_assert_is_up();
- Assert(!IsTransactionOrTransactionBlock());
+ /*
+ * "force" is required if this routine is called inside a transaction
+ * block.
+ */
+ Assert(!is_xact || force);
/* "absorb" the forced flush even if there's nothing to flush */
if (pgStatForceNextFlush)
@@ -789,6 +797,11 @@ pgstat_report_stat(bool force)
if (!kind_info->flush_static_cb)
continue;
+ /* Skip if this stats kind cannot be flushed in a transaction */
+ if (is_xact &&
+ (kind_info->report_context & PGSTAT_REPORT_TRANSACTION) == 0)
+ continue;
+
partial_flush |= kind_info->flush_static_cb(nowait);
}
}
@@ -801,8 +814,11 @@ pgstat_report_stat(bool force)
*/
if (partial_flush)
{
- /* force should have prevented us from getting here */
- Assert(!force);
+ /*
+ * force should have prevented us from getting here, and partial
+ * flushes are accepted inside a transaction.
+ */
+ Assert(!force || is_xact);
/* remember since when stats have been pending */
if (pending_since == 0)
@@ -1351,6 +1367,7 @@ pgstat_flush_pending_entries(bool nowait)
{
bool have_pending = false;
dlist_node *cur = NULL;
+ bool is_xact = IsTransactionOrTransactionBlock();
/*
* Need to be a bit careful iterating over the list of pending entries.
@@ -1377,6 +1394,21 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip if this stats kind cannot be flushed while in a transaction */
+ if (is_xact &&
+ (kind_info->report_context & PGSTAT_REPORT_TRANSACTION) == 0)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index bad5642d9c90..b27a5eb58e18 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -28,6 +28,7 @@
#include "replication/logicallauncher.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "storage/procsignal.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/timestamp.h"
@@ -2325,3 +2326,60 @@ pg_stat_have_stats(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(pgstat_have_entry(kind, dboid, objid));
}
+
+/*
+ * pg_stat_report
+ * Signal a backend or an auxiliary process to have it push an update
+ * of its statistics data.
+ *
+ * By default, only superusers are allowed to signal to log the memory
+ * contexts because allowing any users to issue this request at an unbounded
+ * rate would cause lots of log messages and which can lead to denial of
+ * service. Additional roles can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to update their statistics.
+ */
+Datum
+pg_stat_report(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ proc = AuxiliaryPidGetProc(pid);
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; but by the time we reach kill(), a process for which we
+ * get a valid proc here might have terminated on its own. This is OK,
+ * as at shutdown processes flush their stats.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+ if (SendProcSignal(pid, PROCSIG_STATS_UPDATE, procNumber) < 0)
+ {
+ /* Again, just a warning to allow loops */
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ PG_RETURN_BOOL(true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b3602..22960ee3b27b 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -38,6 +38,7 @@ volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalStatsUpdatePending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 6c03b1a79d75..68a0e1e02bcc 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -397,6 +397,58 @@ REVOKE EXECUTE ON FUNCTION pg_log_backend_memory_contexts(integer)
FROM regress_log_memory;
DROP ROLE regress_log_memory;
--
+-- pg_stat_report
+--
+-- check execution
+SELECT pg_stat_report(pg_backend_pid());
+ pg_stat_report
+----------------
+ t
+(1 row)
+
+SELECT pg_stat_report(pid) FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer';
+ pg_stat_report
+----------------
+ t
+(1 row)
+
+-- Check privileges
+CREATE ROLE regress_stat_report;
+SELECT has_function_privilege('regress_stat_report',
+ 'pg_stat_report(integer)', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+-- Fails
+SET ROLE regress_stat_report;
+SELECT pg_stat_report(pg_backend_pid());
+ERROR: permission denied for function pg_stat_report
+RESET ROLE;
+-- Access granted, then function works
+GRANT EXECUTE ON FUNCTION pg_stat_report(integer)
+ TO regress_stat_report;
+SELECT has_function_privilege('regress_stat_report',
+ 'pg_stat_report(integer)', 'EXECUTE'); -- yes
+ has_function_privilege
+------------------------
+ t
+(1 row)
+
+SET ROLE regress_stat_report;
+SELECT pg_stat_report(pg_backend_pid());
+ pg_stat_report
+----------------
+ t
+(1 row)
+
+RESET ROLE;
+REVOKE EXECUTE ON FUNCTION pg_stat_report(integer)
+ FROM regress_stat_report;
+DROP ROLE regress_stat_report;
+--
-- Test some built-in SRFs
--
-- The outputs of these are variable, so we can't just print their results
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 35b7983996c4..fe366904c9e8 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -154,6 +154,35 @@ REVOKE EXECUTE ON FUNCTION pg_log_backend_memory_contexts(integer)
DROP ROLE regress_log_memory;
+--
+-- pg_stat_report
+--
+
+-- check execution
+SELECT pg_stat_report(pg_backend_pid());
+SELECT pg_stat_report(pid) FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer';
+
+-- Check privileges
+CREATE ROLE regress_stat_report;
+SELECT has_function_privilege('regress_stat_report',
+ 'pg_stat_report(integer)', 'EXECUTE'); -- no
+-- Fails
+SET ROLE regress_stat_report;
+SELECT pg_stat_report(pg_backend_pid());
+RESET ROLE;
+-- Access granted, then function works
+GRANT EXECUTE ON FUNCTION pg_stat_report(integer)
+ TO regress_stat_report;
+SELECT has_function_privilege('regress_stat_report',
+ 'pg_stat_report(integer)', 'EXECUTE'); -- yes
+SET ROLE regress_stat_report;
+SELECT pg_stat_report(pg_backend_pid());
+RESET ROLE;
+REVOKE EXECUTE ON FUNCTION pg_stat_report(integer)
+ FROM regress_stat_report;
+DROP ROLE regress_stat_report;
+
--
-- Test some built-in SRFs
--
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 210b1118bdf7..114202c4fc19 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -220,6 +220,22 @@
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_stat_report</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Requests to update the statistics computed by the backend with the
+ specified process ID. This function can send the request to
+ backends and auxiliary processes except logger, pushing an
+ update of the statistics data.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
--
2.53.0
[application/pgp-signature] signature.asc (832B, ../../abeimWKYq-n34FrH@paquier.xyz/3-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-24 12:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 06:26 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
@ 2026-03-16 09:20 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 10:22 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-03-16 09:20 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Mon, Mar 16, 2026 at 03:26:33PM +0900, Michael Paquier wrote:
> On Tue, Feb 24, 2026 at 12:01:30PM +0000, Bertrand Drouvot wrote:
> > Though I don't think that adresses Michael's concern: "main worries are
> > mainly around 1), I guess, with the new SIGALRM handler requirements for all
> > auxiliary processes" in [1].
>
> FWIW, I am still concerned about that, and I have pondered about what
> we could do here. While reviewing the existing code, one thing that I
> have noticed we could do is rely on the existing interface of
> pgstat_report_stat() without changing the existing callers, and not
> touching at all the flush callbacks. If we begin to require the
> "force" mode when the routine the called inside a transaction block,
> things seem to work pretty smoothly in combination with a stats kind
> property that allows the stats data to be flushed if we are inside a
> transaction while a report happens.
Yeah, "force" makes use of GetCurrentTimestamp() (and so we avoid a failed
assertion that we would get if using GetCurrentTransactionStopTimestamp()).
> So please find attached my shot at that:
Thanks!
> - Introduction of a new system function called pg_stat_report(), based
> on a procsignal that gives a way to signal backends for a stats
> update, reusing the existing code where we only do flushes when idle
> and not in a transaction.
> - Property that tracks under which contexts the reports are allowed.
> Here I have decided to stick with simple, as in only allowing IO and
> WAL stats to be flushed if we are inside a transaction.
>
> Using that, I have done a few tests with three backends:
> - One with a long-running transaction.
> - One that periodically triggers the reports.
> - One that looks at IO and WAL stat.
> And the third session is able to get refreshes for both of these stats
> kinds, while the other stats remain the same.
I did not look closely at the code but did some testing too. I confirm that
pg_stat_io and pg_stat_wal are updated when pg_stat_report(<backend_pid>) is
triggered. But the stats update is not visible if requested through
pg_stat_get_backend_io(<same_backend_pid>) or pg_stat_get_backend_wal(<same_backend_pid>)).
I guess that PGSTAT_KIND_BACKEND should also get the PGSTAT_REPORT_TRANSACTION
report_context?
> Note that this is a WIP, which is check-world stable. One thing that
> sticks a bit in mind now is that perhaps we should not allow the
> function for auxiliary processes at all.
Why?
> A second thing is the
> requirement of allowing partial flushes at the end of the report path,
> which is OK because the variable-sized stats can have pending data.
Right.
> Perhaps we should just have pgstat_flush_pending_entries() provide a
> correct status in line with the property set in a stats kind when we
> try a flush while in a transaction.
The idea would be to avoid trying to flush stats that don't have pending
entries?
> Thoughts or tomatoes?
That looks "simpler" that the previous proposal but who would be responsible to
call pg_stat_report()? If that's the client responsabilty that kind of look weird
to me. If that's the core, how would that be scheduled? I think that the
end solution should prevent to find similar issues as 039549d70f6 fixed, without
delegating to the client.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-24 12:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 06:26 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-03-16 09:20 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-03-16 10:22 ` Michael Paquier <michael@paquier.xyz>
2026-03-16 23:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Michael Paquier @ 2026-03-16 10:22 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Mon, Mar 16, 2026 at 09:20:41AM +0000, Bertrand Drouvot wrote:
> I did not look closely at the code but did some testing too. I confirm that
> pg_stat_io and pg_stat_wal are updated when pg_stat_report(<backend_pid>) is
> triggered. But the stats update is not visible if requested through
> pg_stat_get_backend_io(<same_backend_pid>) or pg_stat_get_backend_wal(<same_backend_pid>)).
> I guess that PGSTAT_KIND_BACKEND should also get the PGSTAT_REPORT_TRANSACTION
> report_context?
Yes, I guess that the transaction-level flag should be set as well for
the backend stats, due to the fact that WAL and IO stats have it set.
>> Note that this is a WIP, which is check-world stable. One thing that
>> sticks a bit in mind now is that perhaps we should not allow the
>> function for auxiliary processes at all.
>
> Why?
I was feeling that there would be an issue with that, and I did not
test it in details yet. For the checkpointer with a busy activity,
the IO stats could be interesting to see in live.
>> Perhaps we should just have pgstat_flush_pending_entries() provide a
>> correct status in line with the property set in a stats kind when we
>> try a flush while in a transaction.
>
> The idea would be to avoid trying to flush stats that don't have pending
> entries?
I had to work around the assert at the end of pgstat_report_stat(), to
tell that under an xact the partial flushes were OK. I was wondering
about keeping the end assertion based on solely "force" intact, making
the flush routines return a status based on if we are in an xact.
> That looks "simpler" that the previous proposal but who would be responsible to
> call pg_stat_report()? If that's the client responsabilty that kind of look weird
> to me. If that's the core, how would that be scheduled? I think that the
> end solution should prevent to find similar issues as 039549d70f6 fixed, without
> delegating to the client.
TBH, I don't like the requirement of setting SIGALRM in all the places
where we'd require it for the sake of this proposal, where
historically we have never done that, copying a mechanism that already
exists in the tree for procsigs, as the previous patch I posted proves
we could reuse. It's also not clear to me what a correct frequency of
the stat updates should be, and why it would make sense to force that
in a GUC; we want to have some information from long-running
transactions, where I fear that we will want modularity. A
user-settable GUC would fit with this picture, but the requirement of
planting the timeouts don't stick for me..
On top of that, I am not really convinced that there is a good reason
to remove the existing stat report calls we have already planted in
the tree for auxiliary processes, diverging from the stable branches
where these exist. With more than one release already out with them,
there is more benefit with potentially planting more strategic report
calls where they could matter (as added in v18 for WAL senders as one
example), when we find a requirement for them.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../abfZ55jwCGGci6aK@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-24 12:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 06:26 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-03-16 09:20 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 10:22 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
@ 2026-03-16 23:42 ` Sami Imseih <samimseih@gmail.com>
2026-03-25 03:16 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 71+ messages in thread
From: Sami Imseih @ 2026-03-16 23:42 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
I have spent some time thinking about Michael's feedback and I want to
see if I can propose an alternate design that we can come to a consensus
on.
> I had to work around the assert at the end of pgstat_report_stat(), to
> tell that under an xact the partial flushes were OK. I was wondering
> about keeping the end assertion based on solely "force" intact, making
> the flush routines return a status based on if we are in an xact.
>
> > That looks "simpler" that the previous proposal but who would be responsible to
> > call pg_stat_report()? If that's the client responsibility that kind of look weird
> > to me. If that's the core, how would that be scheduled? I think that the
> > end solution should prevent to find similar issues as 039549d70f6 fixed, without
> > delegating to the client.
>
> TBH, I don't like the requirement of setting SIGALRM in all the places
> where we'd require it for the sake of this proposal, where
> historically we have never done that, copying a mechanism that already
> exists in the tree for procsigs, as the previous patch I posted proves
> we could reuse.
How about we add a timeout when a transaction goes idle?
Currently we have IDLE_STATS_UPDATE_TIMEOUT, which is the
timeout interval used when we have done more than one flush
within a second. This only works when the session is idle
(not in a transaction), and flushes everything.
However, as I mentioned earlier in this thread, we can also schedule
a flush that occurs when a transaction goes idle. This means when
a transaction is started (BEGIN/START TRANSACTION) the proceeding
idle-in-transaction can schedule a timeout of 10 seconds based on
a new #define called PGSTAT_IDLE_TXN_INTERVAL.
The timeout is not a fixed repeating interval. It is set
on-demand each time the transaction enters idle-in-transaction
state: first after the BEGIN, then again after each subsequent
command completes. This is a trade-off: if a single command
runs for a long time, no flush happens until it finishes and
the transaction goes idle again.
So, we still introduce a new timeout in this mechanism, but
it does not need to be registered all over the place.
So with this all the "safe to be flushed" stats will be flushed
mid-transaction at some point, and all stats will be flushed
at the end of a transaction.
With that said, the only way I can conceive getting away from a
a new timeout in this design is to track timestamp whenever
we go into idle-intransaction, but I am not too comfy adding a
GetCurrentTimestamp() there.
-> if no action occurs at this point, we don't set another timeout
> It's also not clear to me what a correct frequency of
> the stat updates should be, and why it would make sense to force that
> in a GUC;
No GUC will be needed in what I am proposing. A 10s flush interval
should be sufficient.
> On top of that, I am not really convinced that there is a good reason
> to remove the existing stat report calls we have already planted in
> the tree for auxiliary processes, diverging from the stable branches
> where these exist.
The design being proposed keeps all the existing in-tact.
So attached is a new proposal with tests and docs. In terms of
test I fell back to the strategy used by Bertrand [0] with the
INJECTION_POINT trick to reduce the flush interval. It does mean
we keep a pg_sleep in the test, but I think we should not try
to do anything different here.
Also, with this design we can call the flush modes FLUSH_IN_TRANSACTION
and FLUSH_AT_TXN_BOUNDARY. As the earlier proposals, the
callbacks will still need to make the decision what stats to flush based on
if it's called in-transaction or at boundary.
[0] https://www.postgresql.org/message-id/aZbDYMrOkeCyIubO%40ip-10-97-1-34.eu-west-3.compute.internal
--
Sami Imseih
Amazon Web Services (AWS)
Attachments:
[application/octet-stream] v12-0001-Add-periodic-in-transaction-stats-flushing.patch (54.2K, ../../CAA5RZ0ssPraeF9gg8S8G7O25t90t8_Y2PU-jdNkZ42EiNH22CA@mail.gmail.com/2-v12-0001-Add-periodic-in-transaction-stats-flushing.patch)
download | inline diff:
From 5d75d93701bd268e7689791042405450297fede9 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v12 1/1] Add periodic in-transaction stats flushing
Some statistics, such as relation scan counts, blocks fetched/hit,
WAL, IO, and SLRU stats, can safely be flushed to shared memory
mid-transaction. Others, like tuple insert/update/delete counts,
should only be flushed at transaction boundaries.
Introduce a flush_mode field (FLUSH_IN_TRANSACTION and
FLUSH_AT_TXN_BOUNDARY) fo each stats kind. FLUSH_IN_TRANSACTION will
be flushed every PGSTAT_IDLE_TXN_INTERVAL (10s) while idle in transaction.
The timeout is set each time the transaction goes idle between
commands, not on a fixed repeating interval. It is disabled when
the session goes idle after the transaction ends.
---
doc/src/sgml/monitoring.sgml | 28 ++++-
src/backend/replication/walsender.c | 8 +-
src/backend/tcop/postgres.c | 43 +++++++
src/backend/utils/activity/pgstat.c | 109 ++++++++++++++---
src/backend/utils/activity/pgstat_backend.c | 6 +-
src/backend/utils/activity/pgstat_bgwriter.c | 2 +-
.../utils/activity/pgstat_checkpointer.c | 2 +-
src/backend/utils/activity/pgstat_database.c | 6 +-
src/backend/utils/activity/pgstat_function.c | 8 +-
src/backend/utils/activity/pgstat_io.c | 6 +-
src/backend/utils/activity/pgstat_relation.c | 98 ++++++++++++---
src/backend/utils/activity/pgstat_slru.c | 2 +-
.../utils/activity/pgstat_subscription.c | 8 +-
src/backend/utils/activity/pgstat_wal.c | 10 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 12 ++
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 8 ++
src/include/utils/pgstat_internal.h | 58 +++++++--
src/include/utils/timeout.h | 1 +
.../test_custom_stats/t/001_custom_stats.pl | 59 +++++++++
.../test_custom_var_stats--1.0.sql | 10 ++
.../test_custom_stats/test_custom_var_stats.c | 114 +++++++++++++++++-
src/test/modules/test_misc/meson.build | 1 +
.../test_misc/t/011_in_transaction_stats.pl | 91 ++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
26 files changed, 611 insertions(+), 82 deletions(-)
create mode 100644 src/test/modules/test_misc/t/011_in_transaction_stats.pl
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 462019a972c..dcdeb9aabec 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -235,15 +235,31 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
When using the cumulative statistics views and functions to monitor
collected data, it is important to realize that the information does not
update instantaneously. Each individual server process flushes out
- accumulated statistics to shared memory just before going idle, but not
- more frequently than once per <varname>PGSTAT_MIN_INTERVAL</varname>
- milliseconds (1 second unless altered while building the server); so a
- query or transaction still in progress does not affect the displayed totals
- and the displayed information lags behind actual activity. However,
- current-query information collected by <varname>track_activities</varname>
+ accumulated statistics to shared memory just before going idle or
+ periodically during transactions, depending on whether the statistics
+ should only be flushed at a transaction boundary, as described below.
+ Current-query information collected by <varname>track_activities</varname>
is always up-to-date.
</para>
+ <para>
+ When a server process is about to become idle, it flushes all pending
+ statistics to shared memory, but not more frequently than once per
+ <varname>PGSTAT_MIN_INTERVAL</varname> milliseconds (1 second unless
+ altered while building the server).
+ </para>
+
+ <para>
+ During explicit transaction blocks, statistics that can be flushed at any
+ time, such as relation scan counts, tuples returned and fetched, blocks
+ fetched and hit, as well as IO, WAL, SLRU, and backend statistics, are
+ periodically flushed to shared memory every
+ <varname>PGSTAT_IDLE_TXN_INTERVAL</varname> milliseconds (10 seconds
+ unless altered while building the server). This periodic flushing occurs
+ when the transaction goes idle between commands. It does not apply to
+ single statements running outside an explicit transaction block.
+ </para>
+
<para>
Another important point is that when a server process is asked to display
any of the accumulated statistics, accessed values are cached until the end
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 376ff46340d..24124d478e8 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1993,8 +1993,8 @@ WalSndWaitForWal(XLogRecPtr loc)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
@@ -3022,8 +3022,8 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index b3563113219..00436fee0de 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3565,6 +3565,12 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ if (InTransactionStatsUpdateTimeoutPending)
+ {
+ InTransactionStatsUpdateTimeoutPending = false;
+ pgstat_report_in_transaction_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
@@ -4621,6 +4627,21 @@ PostgresMain(const char *dbname, const char *username)
enable_timeout_after(IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
IdleInTransactionSessionTimeout);
}
+
+ /*
+ * Flush in-transaction stats.
+ */
+ if (!get_timeout_active(IN_TRANSACTION_STATS_UPDATE_TIMEOUT))
+ {
+ int stats_interval = PGSTAT_IDLE_TXN_INTERVAL;
+
+#ifdef USE_INJECTION_POINTS
+ if (IS_INJECTION_POINT_ATTACHED("in-transaction-stats-short-interval"))
+ stats_interval = 1000;
+#endif
+ enable_timeout_after(IN_TRANSACTION_STATS_UPDATE_TIMEOUT,
+ stats_interval);
+ }
}
else if (IsTransactionOrTransactionBlock())
{
@@ -4635,6 +4656,21 @@ PostgresMain(const char *dbname, const char *username)
enable_timeout_after(IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
IdleInTransactionSessionTimeout);
}
+
+ /*
+ * Flush in-transaction stats.
+ */
+ if (!get_timeout_active(IN_TRANSACTION_STATS_UPDATE_TIMEOUT))
+ {
+ int stats_interval = PGSTAT_IDLE_TXN_INTERVAL;
+
+#ifdef USE_INJECTION_POINTS
+ if (IS_INJECTION_POINT_ATTACHED("in-transaction-stats-short-interval"))
+ stats_interval = 1000;
+#endif
+ enable_timeout_after(IN_TRANSACTION_STATS_UPDATE_TIMEOUT,
+ stats_interval);
+ }
}
else
{
@@ -4650,6 +4686,13 @@ PostgresMain(const char *dbname, const char *username)
if (notifyInterruptPending)
ProcessNotifyInterrupt(false);
+ /*
+ * We are no longer in a transaction, so disable the
+ * in-transaction stats flush timeout if it was active.
+ */
+ if (get_timeout_active(IN_TRANSACTION_STATS_UPDATE_TIMEOUT))
+ disable_timeout(IN_TRANSACTION_STATS_UPDATE_TIMEOUT, false);
+
/*
* Check if we need to report stats. If pgstat_report_stat()
* decides it's too soon to flush out pending stats / lock
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..70090565fee 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool in_txn_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool in_txn_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_IN_TRANSACTION,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_IN_TRANSACTION,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_IN_TRANSACTION,
.accessed_across_databases = true,
@@ -436,6 +443,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_IN_TRANSACTION,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +461,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_IN_TRANSACTION,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +479,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_IN_TRANSACTION,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +785,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,7 +1291,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
@@ -1345,9 +1344,15 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If in_txn_only is true, only flushes FLUSH_IN_TRANSACTION entries. For entries
+ * that support it, the callback may flush only non-transactional fields.
+ * This is safe to call inside transactions.
+ *
+ * If in_txn_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool in_txn_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1372,13 +1377,29 @@ pgstat_flush_pending_entries(bool nowait)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in in_txn_only mode */
+ if (in_txn_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait,
+ in_txn_only, &is_partial_flush);
Assert(did_flush || nowait);
@@ -1388,8 +1409,8 @@ pgstat_flush_pending_entries(bool nowait)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
@@ -1402,6 +1423,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If in_txn_only is true, only flushes FLUSH_IN_TRANSACTION stats (safe inside transactions).
+ * If in_txn_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool in_txn_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in in_txn_only mode */
+ if (in_txn_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait, in_txn_only);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2167,24 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_IN_TRANSACTION stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ *
+ * This is called from a timeout handler every PGSTAT_IDLE_TXN_INTERVAL
+ * (10000ms) while idle in transaction, to ensure non-transactional stats
+ * (e.g., IO, backend) are flushed periodically even during long transactions.
+ */
+void
+pgstat_report_in_transaction_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index f2f8d3ff75f..3be8b2705b1 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -268,7 +268,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
* if some statistics could not be flushed due to lock contention.
*/
bool
-pgstat_flush_backend(bool nowait, bits32 flags)
+pgstat_flush_backend(bool nowait, bits32 flags, bool in_txn_only)
{
PgStat_EntryRef *entry_ref;
bool has_pending_data = false;
@@ -311,9 +311,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
* If some stats could not be flushed due to lock contention, return true.
*/
bool
-pgstat_backend_flush_cb(bool nowait)
+pgstat_backend_flush_cb(bool nowait, bool in_txn_only)
{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
+ return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, in_txn_only);
}
/*
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index ed2fd801189..1c5f0c3ec40 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -61,7 +61,7 @@ pgstat_report_bgwriter(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 1f70194b7a7..2d89a082464 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -68,7 +68,7 @@ pgstat_report_checkpointer(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 933dcb5cae5..72d757a6631 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -435,7 +435,8 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
@@ -443,6 +444,9 @@ pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
pendingent = (PgStat_StatDBEntry *) entry_ref->pending;
sharedent = (PgStatShared_Database *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index e6b84283c6c..d51371d7967 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,14 +190,20 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
+ Assert(!in_txn_only);
+
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..ec541bf3f78 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -172,9 +172,9 @@ pgstat_fetch_stat_io(void)
* Simpler wrapper of pgstat_io_flush_cb()
*/
void
-pgstat_flush_io(bool nowait)
+pgstat_flush_io(bool nowait, bool in_txn_only)
{
- (void) pgstat_io_flush_cb(nowait);
+ (void) pgstat_io_flush_cb(nowait, in_txn_only);
}
/*
@@ -186,7 +186,7 @@ pgstat_flush_io(bool nowait)
* acquired. Otherwise, return false.
*/
bool
-pgstat_io_flush_cb(bool nowait)
+pgstat_io_flush_cb(bool nowait, bool in_txn_only)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..33359cf07b0 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -47,7 +47,19 @@ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_lev
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
+static void flush_relation_in_transaction_stats(PgStat_StatTabEntry *tabentry,
+ PgStat_TableCounts *counts, bool in_txn_only);
+/*
+ * Update database statistics with non-transactional stats.
+ */
+#define UPDATE_DATABASE_IN_TRANSACTION_STATS(dbentry, counts) \
+ do { \
+ (dbentry)->tuples_returned += (counts)->tuples_returned; \
+ (dbentry)->tuples_fetched += (counts)->tuples_fetched; \
+ (dbentry)->blocks_fetched += (counts)->blocks_fetched; \
+ (dbentry)->blocks_hit += (counts)->blocks_hit; \
+ } while (0)
/*
* Copy stats between relations. This is used for things like REINDEX
@@ -267,8 +279,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* is done -- which will likely vacuum many relations -- or until the
* VACUUM command has processed all tables and committed.
*/
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -362,8 +374,8 @@ pgstat_report_analyze(Relation rel,
pgstat_unlock_entry(entry_ref);
/* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -802,6 +814,29 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Helper function to flush non-transactional statistics.
+ */
+static void
+flush_relation_in_transaction_stats(PgStat_StatTabEntry *tabentry, PgStat_TableCounts *counts,
+ bool in_txn_only)
+{
+ TimestampTz t;
+
+ tabentry->numscans += counts->numscans;
+ if (counts->numscans)
+ {
+ t = in_txn_only ? GetCurrentTimestamp() : GetCurrentTransactionStopTimestamp();
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+
+ tabentry->tuples_returned += counts->tuples_returned;
+ tabentry->tuples_fetched += counts->tuples_fetched;
+ tabentry->blocks_fetched += counts->blocks_fetched;
+ tabentry->blocks_hit += counts->blocks_hit;
+}
+
/*
* Flush out pending stats for the entry
*
@@ -810,9 +845,17 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
*
* Some of the stats are copied to the corresponding pending database stats
* entry when successfully flushing.
+ *
+ * If in_txn_only is true, only non-transactional fields are flushed
+ * (numscans, tuples_returned, tuples_fetched, blocks_fetched, blocks_hit).
+ * Transactional fields remain pending until transaction boundary.
+ *
+ * Some of the stats are copied to the corresponding pending database stats
+ * entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -824,6 +867,9 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+ /* this is a partial flush if in in_txn_only mode */
+ *is_partial = in_txn_only;
+
/*
* Ignore entries that didn't accumulate any actual counts, such as
* indexes that were opened by the planner but not used.
@@ -835,19 +881,36 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
- /* add the values to the shared entry. */
tabentry = &shtabstats->stats;
- tabentry->numscans += lstats->counts.numscans;
- if (lstats->counts.numscans)
+ if (in_txn_only)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
- if (t > tabentry->lastscan)
- tabentry->lastscan = t;
+ /* Flush non-transactional statistics */
+ flush_relation_in_transaction_stats(tabentry, &lstats->counts, true);
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ UPDATE_DATABASE_IN_TRANSACTION_STATS(dbentry, &lstats->counts);
+
+ /*
+ * Clear the flushed fields from pending stats to prevent
+ * double-counting when we flush all fields at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
}
- tabentry->tuples_returned += lstats->counts.tuples_returned;
- tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+
+ /* Flush non-transactional statistics */
+ flush_relation_in_transaction_stats(tabentry, &lstats->counts, false);
+
tabentry->tuples_inserted += lstats->counts.tuples_inserted;
tabentry->tuples_updated += lstats->counts.tuples_updated;
tabentry->tuples_deleted += lstats->counts.tuples_deleted;
@@ -877,9 +940,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
*/
tabentry->ins_since_vacuum += lstats->counts.tuples_inserted;
- tabentry->blocks_fetched += lstats->counts.blocks_fetched;
- tabentry->blocks_hit += lstats->counts.blocks_hit;
-
/* Clamp live_tuples in case of negative delta_live_tuples */
tabentry->live_tuples = Max(tabentry->live_tuples, 0);
/* Likewise for dead_tuples */
@@ -889,13 +949,11 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
/* The entry was successfully flushed, add the same to database stats */
dbentry = pgstat_prep_database_pending(dboid);
- dbentry->tuples_returned += lstats->counts.tuples_returned;
- dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ UPDATE_DATABASE_IN_TRANSACTION_STATS(dbentry, &lstats->counts);
+
dbentry->tuples_inserted += lstats->counts.tuples_inserted;
dbentry->tuples_updated += lstats->counts.tuples_updated;
dbentry->tuples_deleted += lstats->counts.tuples_deleted;
- dbentry->blocks_fetched += lstats->counts.blocks_fetched;
- dbentry->blocks_hit += lstats->counts.blocks_hit;
return true;
}
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..d8dc7be11ef 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -139,7 +139,7 @@ pgstat_get_slru_index(const char *name)
* acquired. Otherwise return false.
*/
bool
-pgstat_slru_flush_cb(bool nowait)
+pgstat_slru_flush_cb(bool nowait, bool in_txn_only)
{
PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
int i;
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 3277cf88a4e..4f289683d33 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -117,14 +117,20 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
+ Assert(!in_txn_only);
+
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
/* localent always has non-zero content */
if (!pgstat_lock_entry(entry_ref, nowait))
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..bf796ad655e 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -51,12 +51,12 @@ pgstat_report_wal(bool force)
nowait = !force;
/* flush wal stats */
- (void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
+ (void) pgstat_wal_flush_cb(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
/* flush IO stats */
- pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -88,7 +88,7 @@ pgstat_wal_have_pending(void)
* acquired. Otherwise return false.
*/
bool
-pgstat_wal_flush_cb(bool nowait)
+pgstat_wal_flush_cb(bool nowait, bool in_txn_only)
{
PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
WalUsage wal_usage_diff = {0};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..bcce54367d5 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t InTransactionStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 26118661f07..27ecf0df16e 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -65,6 +65,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -89,6 +90,7 @@ static void IdleInTransactionSessionTimeoutHandler(void);
static void TransactionTimeoutHandler(void);
static void IdleSessionTimeoutHandler(void);
static void IdleStatsUpdateTimeoutHandler(void);
+static void InTransactionStatsUpdateTimeoutHandler(void);
static void ClientCheckTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
@@ -774,6 +776,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(IN_TRANSACTION_STATS_UPDATE_TIMEOUT,
+ InTransactionStatsUpdateTimeoutHandler);
}
/*
@@ -1433,6 +1437,14 @@ IdleStatsUpdateTimeoutHandler(void)
SetLatch(MyLatch);
}
+static void
+InTransactionStatsUpdateTimeoutHandler(void)
+{
+ InTransactionStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
+
static void
ClientCheckTimeoutHandler(void)
{
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..7051e4b6f3f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t InTransactionStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 216b93492ba..8571eb1afd7 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -38,6 +38,13 @@ typedef struct RelationData *Relation;
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/*
+ * Interval in milliseconds for flushing FLUSH_IN_TRANSACTION stats to shared
+ * memory. An explicit transaction always enters idle-in-transaction state
+ * between commands, which is when this timeout is enabled.
+ */
+#define PGSTAT_IDLE_TXN_INTERVAL 10000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -536,6 +543,7 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_in_transaction_stat(bool force);
extern void pgstat_force_next_flush(void);
extern void pgstat_reset_counters(void);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..e9f4dc85528 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,18 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_IN_TRANSACTION, /* All fields can be flushed in a transaction */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +263,16 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * The mode of when to flush stats. See PgStat_FlushMode for more details.
+ *
+ * This member only has meaning for statistics kinds that accumulate
+ * pending stats and use flush callbacks. For kinds that write directly to
+ * shared memory (e.g., archiver, bgwriter, checkpointer), this member has
+ * no effect.
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -297,8 +319,13 @@ typedef struct PgStat_KindInfo
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
+ *
+ * The in_txn_only parameter indicates whether this is an in-transaction
+ * flush. The is_partial parameter indicates whether this is a partial
+ * flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait,
+ bool in_txn_only, bool *is_partial);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -366,8 +393,11 @@ typedef struct PgStat_KindInfo
*
* "pgstat_report_fixed" needs to be set to trigger the flush of pending
* stats.
+ *
+ * The in_txn_only parameter indicates whether this is an in-transaction
+ * flush.
*/
- bool (*flush_static_cb) (bool nowait);
+ bool (*flush_static_cb) (bool nowait, bool in_txn_only);
/*
* For fixed-numbered statistics: Reset All.
@@ -696,8 +726,8 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
-extern bool pgstat_flush_backend(bool nowait, bits32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
+extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool in_txn_only);
+extern bool pgstat_backend_flush_cb(bool nowait, bool in_txn_only);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -729,7 +759,8 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -737,7 +768,8 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -745,9 +777,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_io.c
*/
-extern void pgstat_flush_io(bool nowait);
+extern void pgstat_flush_io(bool nowait, bool in_txn_only);
-extern bool pgstat_io_flush_cb(bool nowait);
+extern bool pgstat_io_flush_cb(bool nowait, bool in_txn_only);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -762,7 +794,8 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -809,7 +842,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
* Functions in pgstat_slru.c
*/
-extern bool pgstat_slru_flush_cb(bool nowait);
+extern bool pgstat_slru_flush_cb(bool nowait, bool in_txn_only);
extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -820,7 +853,7 @@ extern void pgstat_slru_snapshot_cb(void);
*/
extern void pgstat_wal_init_backend_cb(void);
-extern bool pgstat_wal_flush_cb(bool nowait);
+extern bool pgstat_wal_flush_cb(bool nowait, bool in_txn_only);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
@@ -830,7 +863,8 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..68fe814bde6 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -34,6 +34,7 @@ typedef enum TimeoutId
TRANSACTION_TIMEOUT,
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
+ IN_TRANSACTION_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 9e6a7a38577..d3afbff37c5 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -156,5 +156,64 @@ $result = $node->safe_psql('postgres',
);
is($result, "0", "report of fixed-sized after manual reset");
+# Test FLUSH_IN_TRANSACTION custom stats kind.
+# The intxn kind is registered alongside the regular kind by test_custom_var_stats.
+
+# Basic update and report.
+$node->safe_psql('postgres',
+ q(SELECT test_custom_var_intxn_update('intxn_entry1')));
+$node->safe_psql('postgres',
+ q(SELECT test_custom_var_intxn_update('intxn_entry1')));
+$node->safe_psql('postgres',
+ q(SELECT test_custom_var_intxn_update('intxn_entry1')));
+
+$result = $node->safe_psql('postgres',
+ q(SELECT test_custom_var_intxn_report('intxn_entry1')));
+is($result, "3", "intxn stats report after updates");
+
+# Verify intxn stats are not persisted (write_to_file = false).
+$node->stop();
+$node->start();
+
+$result = $node->safe_psql('postgres',
+ q(SELECT test_custom_var_intxn_report('intxn_entry1')));
+is($result, "", "intxn stats lost after clean restart (not persisted)");
+
+# Test in-transaction flushing with injection point.
+SKIP:
+{
+ skip "injection points not supported", 1
+ unless (($ENV{enable_injection_points} // '') eq 'yes'
+ && $node->check_extension('injection_points'));
+
+ $node->safe_psql('postgres', 'CREATE EXTENSION IF NOT EXISTS injection_points;');
+ $node->safe_psql('postgres',
+ "SELECT injection_points_attach('in-transaction-stats-short-interval', 'error');");
+
+ $node->append_conf('postgresql.conf', 'stats_fetch_consistency = none');
+ $node->reload;
+
+ my $intxn_before = $node->safe_psql('postgres',
+ "SELECT COALESCE(test_custom_var_intxn_report('intxn_flush'), 0);");
+
+ $result = $node->safe_psql('postgres', q{
+BEGIN;
+SELECT test_custom_var_intxn_update('intxn_flush');
+SELECT test_custom_var_intxn_update('intxn_flush');
+SELECT test_custom_var_intxn_update('intxn_flush');
+SELECT pg_sleep(2);
+SELECT COALESCE(test_custom_var_intxn_report('intxn_flush'), 0);
+});
+
+ my @lines = split(/\n/, $result);
+ my $intxn_mid_txn = $lines[-1];
+
+ ok($intxn_mid_txn > $intxn_before,
+ "custom intxn stats flushed mid-transaction (before: $intxn_before, mid-txn: $intxn_mid_txn)");
+
+ $node->safe_psql('postgres',
+ "SELECT injection_points_detach('in-transaction-stats-short-interval');");
+}
+
# Test completed successfully
done_testing();
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
index 5ed8cfc2dcf..03650cbc414 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
@@ -24,3 +24,13 @@ CREATE FUNCTION test_custom_stats_var_report(INOUT name TEXT,
RETURNS SETOF record
AS 'MODULE_PATHNAME', 'test_custom_stats_var_report'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_var_intxn_update(IN name TEXT)
+RETURNS void
+AS 'MODULE_PATHNAME', 'test_custom_var_intxn_update'
+LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_var_intxn_report(IN name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'test_custom_var_intxn_report'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 2ef0e903745..e87d47a28ab 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -37,6 +37,11 @@ PG_MODULE_MAGIC_EXT(
*/
#define PGSTAT_KIND_TEST_CUSTOM_VAR_STATS 25
+/*
+ * Kind ID for test_custom_var_intxn_stats statistics (FLUSH_IN_TRANSACTION).
+ */
+#define PGSTAT_KIND_TEST_CUSTOM_VAR_INTXN_STATS 27
+
/* File paths for auxiliary data serialization */
#define TEST_CUSTOM_AUX_DATA_DESC "pg_stat/test_custom_var_stats_desc.stats"
@@ -45,6 +50,8 @@ PG_MODULE_MAGIC_EXT(
*/
#define PGSTAT_CUSTOM_VAR_STATS_IDX(name) hash_bytes_extended((const unsigned char *) name, strlen(name), 0)
+#define PGSTAT_CUSTOM_VAR_INTXN_IDX(name) hash_bytes_extended((const unsigned char *) name, strlen(name), 0)
+
/*--------------------------------------------------------------------------
* Type definitions
*--------------------------------------------------------------------------
@@ -56,6 +63,12 @@ typedef struct PgStat_StatCustomVarEntry
PgStat_Counter numcalls; /* times statistic was incremented */
} PgStat_StatCustomVarEntry;
+/* Pending entry for FLUSH_IN_TRANSACTION kind (same layout) */
+typedef struct PgStat_CustomVarInTxnEntry
+{
+ PgStat_Counter numcalls;
+} PgStat_CustomVarInTxnEntry;
+
/* Shared memory statistics entry visible to all backends */
typedef struct PgStatShared_CustomVarEntry
{
@@ -64,6 +77,13 @@ typedef struct PgStatShared_CustomVarEntry
dsa_pointer description; /* pointer to description string in DSA */
} PgStatShared_CustomVarEntry;
+/* Shared memory entry for FLUSH_IN_TRANSACTION kind */
+typedef struct PgStatShared_CustomVarInTxnEntry
+{
+ PgStatShared_Common header;
+ PgStat_CustomVarInTxnEntry stats;
+} PgStatShared_CustomVarInTxnEntry;
+
/*--------------------------------------------------------------------------
* Global Variables
*--------------------------------------------------------------------------
@@ -85,7 +105,8 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait);
+ bool nowait, bool in_txn_only,
+ bool *is_partial);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -100,6 +121,11 @@ static bool test_custom_stats_var_from_serialized_data(const PgStat_HashKey *key
/* Finish callback: end of statistics file operations */
static void test_custom_stats_var_finish(PgStat_StatsFileOp status);
+/* Flush callback for FLUSH_IN_TRANSACTION kind */
+static bool test_custom_var_intxn_flush_pending_cb(PgStat_EntryRef *entry_ref,
+ bool nowait, bool in_txn_only,
+ bool *is_partial);
+
/*--------------------------------------------------------------------------
* Custom kind configuration
*--------------------------------------------------------------------------
@@ -121,6 +147,19 @@ static const PgStat_KindInfo custom_stats = {
.finish = test_custom_stats_var_finish,
};
+static const PgStat_KindInfo custom_intxn_stats = {
+ .name = "test_custom_var_intxn_stats",
+ .fixed_amount = false,
+ .write_to_file = false, /* no persistence needed for test */
+ .flush_mode = FLUSH_IN_TRANSACTION,
+ .accessed_across_databases = true,
+ .shared_size = sizeof(PgStatShared_CustomVarInTxnEntry),
+ .shared_data_off = offsetof(PgStatShared_CustomVarInTxnEntry, stats),
+ .shared_data_len = sizeof(PgStat_CustomVarInTxnEntry),
+ .pending_size = sizeof(PgStat_CustomVarInTxnEntry),
+ .flush_pending_cb = test_custom_var_intxn_flush_pending_cb,
+};
+
/*--------------------------------------------------------------------------
* Module initialization
*--------------------------------------------------------------------------
@@ -133,8 +172,9 @@ _PG_init(void)
if (!process_shared_preload_libraries_in_progress)
return;
- /* Register custom statistics kind */
+ /* Register custom statistics kinds */
pgstat_register_kind(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, &custom_stats);
+ pgstat_register_kind(PGSTAT_KIND_TEST_CUSTOM_VAR_INTXN_STATS, &custom_intxn_stats);
}
/*--------------------------------------------------------------------------
@@ -152,7 +192,8 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
@@ -160,6 +201,9 @@ test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
shared_entry = (PgStatShared_CustomVarEntry *) entry_ref->shared_stats;
+ /* this is not a partial flush */
+ *is_partial = false;
+
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
@@ -691,3 +735,67 @@ test_custom_stats_var_report(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*--------------------------------------------------------------------------
+ * FLUSH_IN_TRANSACTION kind: callbacks and SQL functions
+ *--------------------------------------------------------------------------
+ */
+
+static bool
+test_custom_var_intxn_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait,
+ bool in_txn_only, bool *is_partial)
+{
+ PgStat_CustomVarInTxnEntry *pending;
+ PgStatShared_CustomVarInTxnEntry *shared;
+
+ pending = (PgStat_CustomVarInTxnEntry *) entry_ref->pending;
+ shared = (PgStatShared_CustomVarInTxnEntry *) entry_ref->shared_stats;
+
+ *is_partial = false;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shared->stats.numcalls += pending->numcalls;
+
+ pgstat_unlock_entry(entry_ref);
+
+ return true;
+}
+
+PG_FUNCTION_INFO_V1(test_custom_var_intxn_update);
+Datum
+test_custom_var_intxn_update(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_EntryRef *entry_ref;
+ PgStat_CustomVarInTxnEntry *pending;
+
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_INTXN_STATS,
+ InvalidOid,
+ PGSTAT_CUSTOM_VAR_INTXN_IDX(stat_name),
+ NULL);
+
+ pending = (PgStat_CustomVarInTxnEntry *) entry_ref->pending;
+ pending->numcalls++;
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(test_custom_var_intxn_report);
+Datum
+test_custom_var_intxn_report(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_CustomVarInTxnEntry *stats;
+
+ stats = (PgStat_CustomVarInTxnEntry *)
+ pgstat_fetch_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_INTXN_STATS,
+ InvalidOid,
+ PGSTAT_CUSTOM_VAR_INTXN_IDX(stat_name));
+
+ if (!stats)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(stats->numcalls);
+}
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index 6e8db1621a7..70248360d21 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -19,6 +19,7 @@ tests += {
't/008_replslot_single_user.pl',
't/009_log_temp_files.pl',
't/010_index_concurrently_upsert.pl',
+ 't/011_in_transaction_stats.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/011_in_transaction_stats.pl b/src/test/modules/test_misc/t/011_in_transaction_stats.pl
new file mode 100644
index 00000000000..cf1bde24a5d
--- /dev/null
+++ b/src/test/modules/test_misc/t/011_in_transaction_stats.pl
@@ -0,0 +1,91 @@
+
+# Copyright (c) 2024-2026, PostgreSQL Global Development Group
+
+# Test in-transaction stats flushing mechanism.
+#
+# This test verifies that FLUSH_IN_TRANSACTION stats are periodically flushed
+# to shared memory while a transaction is idle between commands. Uses an
+# injection point to reduce the flush interval from 10 seconds to 1 second
+# for faster testing.
+#
+# We test one representative of each stats kind: relation stats (seq_scan)
+# for variable-numbered stats, and WAL for fixed-sized stats.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init();
+$node->append_conf('postgresql.conf', 'stats_fetch_consistency = none');
+$node->start;
+
+if (!$node->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attach injection point to reduce the in-transaction stats flush interval
+# to 1 second.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('in-transaction-stats-short-interval', 'error');");
+
+# Create test table
+$node->safe_psql('postgres',
+ 'CREATE TABLE test_in_txn_stats(a int) WITH (autovacuum_enabled = off);');
+$node->safe_psql('postgres',
+ 'INSERT INTO test_in_txn_stats SELECT generate_series(1, 1000);');
+
+# Force flush and get baseline stats
+$node->safe_psql('postgres', 'SELECT pg_stat_force_next_flush();');
+my $seq_scan_before = $node->safe_psql('postgres',
+ "SELECT seq_scan FROM pg_stat_user_tables WHERE relname = 'test_in_txn_stats';");
+
+# Test that seq_scan stats (variable-numbered) are flushed mid-transaction
+# via the in-transaction periodic flush mechanism.
+my $result = $node->safe_psql('postgres', q{
+BEGIN;
+SELECT COUNT(*) FROM test_in_txn_stats;
+SELECT COUNT(*) FROM test_in_txn_stats;
+SELECT pg_sleep(2);
+SELECT seq_scan FROM pg_stat_user_tables WHERE relname = 'test_in_txn_stats';
+});
+
+my @lines = split(/\n/, $result);
+my $seq_scan_mid_txn = $lines[-1];
+
+ok($seq_scan_mid_txn > $seq_scan_before,
+ "seq_scan stats flushed during transaction (before: $seq_scan_before, mid-txn: $seq_scan_mid_txn)");
+
+# Test WAL stats (fixed-sized) flushing during transaction.
+$node->safe_psql('postgres', 'SELECT pg_stat_reset_shared(\'wal\');');
+my $wal_records_before = $node->safe_psql('postgres',
+ "SELECT wal_records FROM pg_stat_wal;");
+
+$result = $node->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_in_txn_stats SELECT generate_series(1, 1000);
+SELECT pg_sleep(2);
+SELECT wal_records FROM pg_stat_wal;
+});
+
+@lines = split(/\n/, $result);
+my $wal_records_mid_txn = $lines[-1];
+
+ok($wal_records_mid_txn > $wal_records_before,
+ "WAL stats flushed during transaction (before: $wal_records_before, mid-txn: $wal_records_mid_txn)");
+
+# Cleanup
+$node->safe_psql('postgres', 'DROP TABLE test_in_txn_stats;');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 52f8603a7be..c9cd6087abc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2290,6 +2290,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.47.3
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-24 12:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 06:26 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-03-16 09:20 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 10:22 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-03-16 23:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-03-25 03:16 ` Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 71+ messages in thread
From: Michael Paquier @ 2026-03-25 03:16 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Mon, Mar 16, 2026 at 06:42:44PM -0500, Sami Imseih wrote:
> So attached is a new proposal with tests and docs. In terms of
> test I fell back to the strategy used by Bertrand [0] with the
As far as I am reading the patch, it seems to me that this would
accelerate the frequency of the stats flushes when we have a
transaction with many short queries, but it does not help much in the
case of an analytical single query because the flush of the stats
would just happen once after the timeout set for each query?
Let's imagine for example a query that takes hours to run, where we'd
want to get fresh IO and WAL stats (backend included) on a periodic
basis by processing interrupts while going through the executor.
Thinking more about this problem, I'd really like to think that a
client-side API would provide a more flexible interface for the
analytical case.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../acNTfL1xO_UUXkZQ@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-23 23:48 ` Michael Paquier <michael@paquier.xyz>
2026-02-24 13:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Michael Paquier @ 2026-02-23 23:48 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Thu, Feb 19, 2026 at 08:01:36AM +0000, Bertrand Drouvot wrote:
> On Thu, Feb 19, 2026 at 12:58:12PM +0900, Michael Paquier wrote:
>> 2) The timeout requirement itself, relying on a timeout threshold
>> controlled by a backend-side configuration.
>
> What are you concerns with this?
I am concerned about the three additional points/requirements:
1) The need for all processes who want to flush non-transactional
stats to set up timeouts, unconditionally, which is what the patch
shows with the new InitializeTimeouts() calls added for example for
auxiliary processes. This forces the use of SIGALRM in these
processes, with a new handler, and feels like a requirement too heavy
for the sake of some stats flushes. This requires an extra
unconditional RegisterTimeout(), as well.
2) The need for all the stats to call pgstat_schedule_anytime_update()
in strategical places. This is less of a burden compared to 1), but
this leads to more complications in these code paths with the coding
requirements, especially for custom stats kinds.
3) Enforcing a flush timeout unconditionally. One thing that slightly
concerns me here is that it seems easier to misuse compared to a
client-based facility, where a too aggressive setup could stress more
the server-side pgstats. It is true that the client-side could be
misused too.
My main worries are mainly around 1), I guess, with the new SIGALRM
handler requirements for all auxiliary processes. Using a procsignal
path would allow us to rely on a solution that has the same
flexibility, combined with strategic additional flush calls that we
could spread depending on requirements we want to enforce in some
processes, like in the WAL sender, or perhaps the checkpointer. That
seems more careful in the long-run, and this can rely on the interrupt
processing for the job. The addition of the property to track if a
stats kind of OK to flush outside a transaction boundary is also
critical to have, of course. I am sold to the point point of the
design about this new property tracked in the stats kind meta-data.
>> With that in mind, wouldn't it be simpler if we introduced an API that
>> could be used from client applications instead, in a model similar
>> what we do for procsignal.c/h?
>
> That's another angle to look at it but I think that giving this responsability to
> the clients would not solve the concerns we had in [1] (that led to 039549d70f6
> and to this thread). It seems to me that a solution/design that does not allow
> us to "revert" 039549d70f6 does not suit our needs. Thoughts?
039549d70f6 goes in line with the "client" prospective, where I would
like to think that strategic flush calls are more flexible.
> Yeah, after our off-list discussion yesterday, I tried to implement the same
> trick that f1e251be80a has done with injection points (nice trick by the way!),
> but that led to:
In this case, avoiding an injpoint allocation in a critical section
would be a two-step process:
- INJECTION_POINT_LOAD(), before the critical section, to warm up the
cache and do all the allocations.
- INJECTION_POINT_CACHED() with IS_INJECTION_POINT_ATTACHED()
(optional), to run the point, in the critical section.
This tactic is already in use in the tree. You would have to use a
more complicated scheme to make sure that the wait machinery is
initialized when using it in a critical section, but that can be
worked around as well. See my "state-of-the-art" work of
15f68cebdcec, which, well, happens to work. Not really the best work
ever with two points, but it works for this purpose. :D
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aZznT84Ssh8PywcH@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:48 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
@ 2026-02-24 13:55 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-24 16:32 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-26 03:33 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-24 13:55 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Tue, Feb 24, 2026 at 08:48:31AM +0900, Michael Paquier wrote:
> On Thu, Feb 19, 2026 at 08:01:36AM +0000, Bertrand Drouvot wrote:
> > On Thu, Feb 19, 2026 at 12:58:12PM +0900, Michael Paquier wrote:
> >> 2) The timeout requirement itself, relying on a timeout threshold
> >> controlled by a backend-side configuration.
> >
> > What are you concerns with this?
>
> I am concerned about the three additional points/requirements:
> 1) The need for all processes who want to flush non-transactional
> stats to set up timeouts, unconditionally, which is what the patch
> shows with the new InitializeTimeouts() calls added for example for
> auxiliary processes. This forces the use of SIGALRM in these
> processes,
Right but they all already call pqsignal(SIGALRM, SIG_IGN), so I'm not sure
to get the point.
> This requires an extra unconditional RegisterTimeout(), as well.
Yes, I'm not clear why that's an issue though.
> 2) The need for all the stats to call pgstat_schedule_anytime_update()
> in strategical places. This is less of a burden compared to 1), but
> this leads to more complications in these code paths with the coding
> requirements, especially for custom stats kinds.
I think that's solved with Sami's proposal for variable stats kind (to flush or
schedule when the session is idle).
> 3) Enforcing a flush timeout unconditionally.
What do you mean? It's done only if there is things to flush.
> My main worries are mainly around 1), I guess, with the new SIGALRM
> handler requirements for all auxiliary processes. Using a procsignal
> path would allow us to rely on a solution that has the same
> flexibility, combined with strategic additional flush calls that we
> could spread depending on requirements we want to enforce in some
> processes, like in the WAL sender, or perhaps the checkpointer.
I see, but we know they'll have to flush IO or WAL stats at some point so
enabling the timeout for those look ok.
> The addition of the property to track if a
> stats kind of OK to flush outside a transaction boundary is also
> critical to have, of course. I am sold to the point point of the
> design about this new property tracked in the stats kind meta-data.
Great!
> 039549d70f6 goes in line with the "client" prospective, where I would
> like to think that strategic flush calls are more flexible.
>
> > Yeah, after our off-list discussion yesterday, I tried to implement the same
> > trick that f1e251be80a has done with injection points (nice trick by the way!),
> > but that led to:
>
> In this case, avoiding an injpoint allocation in a critical section
> would be a two-step process:
> - INJECTION_POINT_LOAD(), before the critical section, to warm up the
> cache and do all the allocations.
> - INJECTION_POINT_CACHED() with IS_INJECTION_POINT_ATTACHED()
> (optional), to run the point, in the critical section.
Oh okay, thanks for the explanation!
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:48 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-24 13:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-24 16:32 ` Sami Imseih <samimseih@gmail.com>
1 sibling, 0 replies; 71+ messages in thread
From: Sami Imseih @ 2026-02-24 16:32 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
> > 2) The need for all the stats to call pgstat_schedule_anytime_update()
> > in strategical places. This is less of a burden compared to 1), but
> > this leads to more complications in these code paths with the coding
> > requirements, especially for custom stats kinds.
>
> I think that's solved with Sami's proposal for variable stats kind (to flush or
> schedule when the session is idle).
Just a clarification. It is "to schedule a flush when the transaction
becomes idle"
PGSTAT_IDLE_INTERVAL already deals when the session is idle.
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:48 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-02-24 13:55 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-26 03:33 ` Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 71+ messages in thread
From: Michael Paquier @ 2026-02-26 03:33 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Tue, Feb 24, 2026 at 01:55:48PM +0000, Bertrand Drouvot wrote:
> On Tue, Feb 24, 2026 at 08:48:31AM +0900, Michael Paquier wrote:
>> I am concerned about the three additional points/requirements:
>> 1) The need for all processes who want to flush non-transactional
>> stats to set up timeouts, unconditionally, which is what the patch
>> shows with the new InitializeTimeouts() calls added for example for
>> auxiliary processes. This forces the use of SIGALRM in these
>> processes,
>
> Right but they all already call pqsignal(SIGALRM, SIG_IGN), so I'm not sure
> to get the point.
This design requires enabling a new signal with a signal handler in a
lot of processes that did not do that. Enabling timeouts in a bunch
of new processes, while claiming it is fine to do, sounds like
something we should rather be careful about. Are you sure for example
that some of the checkpointer code would not buzz on that, for
example?
At the end, this approach seems too heavy-handed to me. I am also not
entirely convinced that enforcing that unconditionally is the right
thing to do in some cases, either. For the bgwriter, as one example,
we have a WAL report call happening in its main loop, which is quite
good in terms of information frequency obtained.
Wouldn't some case-by-case strategic "anytime" API flush calls make
more sense for some subsystems, rather than relying on a timeout? It
seems rather easy to misuse this design in some bgworker contexts, and
SIGALRM could also be used for a different purpose but we would block
that entirely moving on with future versions?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aZ-_AkG3cmeTbdFn@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
@ 2026-01-30 09:46 ` Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 12:42 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
1 sibling, 1 reply; 71+ messages in thread
From: Álvaro Herrera @ 2026-01-30 09:46 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On 2026-Jan-22, Fujii Masao wrote:
> I haven't read the patch in detail yet, but after applying patch 0001 and
> causing a lock wait (for example, using the steps below), I observed that
> log_lock_waits messages are emitted every second.
Interesting. Bertrand asked me about this. He says[1] he diagnosed this
down to SetLatch being called immediately after the handler runs, and I
wonder if it's correct that other timeout handler functions used for
RegisterTimeout are doing SetLatch(). The SIGALRM handler
handle_sig_alarm() that calls the specific handler function already has
a SetLatch call, so why do we need another one here?
I tested this theory quickly by removing the SetLatch from
IdleStatsUpdateTimeoutHandler() and rerunning the tests. (This one
chosen because it has highest coverage per [2]). Everything passed,
though of course this is probably not proof enough.
Maybe we're just cargo-culting these SetLatch() calls?
[1] https://postgr.es/m/aXJUK90lKXw3wrZn@ip-10-97-1-34.eu-west-3.compute.internal
[2] https://coverage.postgresql.org/src/backend/utils/init/postinit.c.gcov.html
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Right now the sectors on the hard disk run clockwise, but I heard a rumor that
you can squeeze 0.2% more throughput by running them counterclockwise.
It's worth the effort. Recommended." (Gerry Pourwelle)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
@ 2026-01-30 12:42 ` Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 14:37 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
0 siblings, 1 reply; 71+ messages in thread
From: Álvaro Herrera @ 2026-01-30 12:42 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On 2026-Jan-30, Álvaro Herrera wrote:
> Interesting. Bertrand asked me about this. He says[1] he diagnosed this
> down to SetLatch being called immediately after the handler runs, and I
> wonder if it's correct that other timeout handler functions used for
> RegisterTimeout are doing SetLatch(). The SIGALRM handler
> handle_sig_alarm() that calls the specific handler function already has
> a SetLatch call, so why do we need another one here?
So apparently the first function to do this in postinit.c was added by
commit c6dda1f48e57 -- and apparently it was mimicking
CheckDeadLockAlert(), which at this time looked like this:
void
CheckDeadLockAlert(void)
{
int save_errno = errno;
got_deadlock_timeout = true;
/*
* Have to set the latch again, even if handle_sig_alarm already did. Back
* then got_deadlock_timeout wasn't yet set... It's unlikely that this
* ever would be a problem, but setting a set latch again is cheap.
*/
SetLatch(MyLatch);
errno = save_errno;
}
I wonder if it would work to move the SetLatch() that's currently at the
top of handle_sig_alarm() to the bottom, after we have called all the
specific handlers, and remove the ones that are there.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 12:42 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
@ 2026-01-30 14:37 ` Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 15:10 ` Re: Flush some statistics within running transactions Andres Freund <andres@anarazel.de>
2026-02-03 06:19 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 2 replies; 71+ messages in thread
From: Álvaro Herrera @ 2026-01-30 14:37 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>; Andres Freund <andres@anarazel.de>
On 2026-Jan-30, Álvaro Herrera wrote:
> So apparently the first function to do this in postinit.c was added by
> commit c6dda1f48e57 -- and apparently it was mimicking
> CheckDeadLockAlert(), which at this time looked like this:
I'm now wondering if CheckDeadLockAlert() really needed to have this in
the first place, or it was just an exercise in paranoia ... it was added
by commit 6753333f55e1, with the discussion in [1], and it's not clear
to me that there was any theoretical or experimental evidence that it
was necessary; the thread didn't discuss it, and the commit message
doesn't either. Added Andres to CC as committer to this thread, maybe
he remembers.
[1] https://www.postgresql.org/message-id/flat/20150115020335.GZ5245%40awork2.anarazel.de
Just for laughs I moved the SetLatch call in handle_sig_alarm() to the
bottom, and remove the ones in handlers, on the theory that by the time
the SetLatch call is reached, all the handlers have already run and thus
the flag variables are set. Everything seems to continue to work:
https://cirrus-ci.com/build/5758839359799296
(Though to be honest, it's not clear to me why it would matter at which
point in handle_sig_alarm we call SetLatch relative to the variables
being set, given that these variables are only going to matter once the
signal handler returns to the original code and the next
CHECK_FOR_INTERRUPTS is hit.)
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Pensar que el espectro que vemos es ilusorio no lo despoja de espanto,
sólo le suma el nuevo terror de la locura" (Perelandra, C.S. Lewis)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 12:42 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 14:37 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
@ 2026-01-30 15:10 ` Andres Freund <andres@anarazel.de>
2026-01-30 16:18 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
1 sibling, 1 reply; 71+ messages in thread
From: Andres Freund @ 2026-01-30 15:10 UTC (permalink / raw)
To: Álvaro Herrera <alvherre@kurilemu.de>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On 2026-01-30 15:37:57 +0100, Álvaro Herrera wrote:
> I'm now wondering if CheckDeadLockAlert() really needed to have this in
> the first place, or it was just an exercise in paranoia ... it was added
> by commit 6753333f55e1, with the discussion in [1], and it's not clear
> to me that there was any theoretical or experimental evidence that it
> was necessary; the thread didn't discuss it, and the commit message
> doesn't either. Added Andres to CC as committer to this thread, maybe
> he remembers.
I don't remember. But back then way more complicated things were still running
in signal handlers, and some signal handlers were capable of interrupting
other signal handlers. Including doing crazy things like starting transactions
in signal handlers (e.g. to process notify interrupts), which in turn could
clear latches. So there was a lot more potential to stomp on each others work.
WRT the subject of this thread: I hope we aren't just enabling a timer to fire
once a second forever but only when there actually is outstanding work?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 12:42 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 14:37 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 15:10 ` Re: Flush some statistics within running transactions Andres Freund <andres@anarazel.de>
@ 2026-01-30 16:18 ` Álvaro Herrera <alvherre@kurilemu.de>
0 siblings, 0 replies; 71+ messages in thread
From: Álvaro Herrera @ 2026-01-30 16:18 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On 2026-Jan-30, Andres Freund wrote:
> I don't remember. But back then way more complicated things were still running
> in signal handlers, and some signal handlers were capable of interrupting
> other signal handlers. Including doing crazy things like starting transactions
> in signal handlers (e.g. to process notify interrupts), which in turn could
> clear latches. So there was a lot more potential to stomp on each others work.
OK, thanks for clarifying. I think my proposal of moving the SetLatch()
needs more research, but it's likely the best way to address this
wrinkle.
> WRT the subject of this thread: I hope we aren't just enabling a timer
> to fire once a second forever but only when there actually is
> outstanding work?
I hope so too. (Just to be clear, I'm not claiming $SUBJECT as its
potential committer, and haven't actually reviewed it.)
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"People get annoyed when you try to debug them." (Larry Wall)
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 12:42 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 14:37 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
@ 2026-02-03 06:19 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-03 17:09 ` Re: Flush some statistics within running transactions Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-03 06:19 UTC (permalink / raw)
To: Álvaro Herrera <alvherre@kurilemu.de>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>; Andres Freund <andres@anarazel.de>
Hi,
On Fri, Jan 30, 2026 at 03:37:57PM +0100, Álvaro Herrera wrote:
> On 2026-Jan-30, Álvaro Herrera wrote:
>
> > So apparently the first function to do this in postinit.c was added by
> > commit c6dda1f48e57 -- and apparently it was mimicking
> > CheckDeadLockAlert(), which at this time looked like this:
>
> I'm now wondering if CheckDeadLockAlert() really needed to have this in
> the first place, or it was just an exercise in paranoia ... it was added
> by commit 6753333f55e1, with the discussion in [1], and it's not clear
> to me that there was any theoretical or experimental evidence that it
> was necessary; the thread didn't discuss it, and the commit message
> doesn't either. Added Andres to CC as committer to this thread, maybe
> he remembers.
>
> [1] https://www.postgresql.org/message-id/flat/20150115020335.GZ5245%40awork2.anarazel.de
>
> Just for laughs I moved the SetLatch call in handle_sig_alarm() to the
> bottom, and remove the ones in handlers, on the theory that by the time
> the SetLatch call is reached, all the handlers have already run and thus
> the flag variables are set. Everything seems to continue to work:
> https://cirrus-ci.com/build/5758839359799296
Thanks for having looked at this!
> (Though to be honest, it's not clear to me why it would matter at which
> point in handle_sig_alarm we call SetLatch relative to the variables
> being set, given that these variables are only going to matter once the
> signal handler returns to the original code and the next
> CHECK_FOR_INTERRUPTS is hit.)
Yeah, I think that we could keep the SetLatch() at the top of handle_sig_alarm().
My understanding is that the signal handler runs to completion without being interrupted
by the code it interrupted. So, by the time the interrupted code (like epoll_wait())
resumes and can check the latch, the entire handler has finished. So, if my understanding
is correct, having SetLatch() at the top or the bottom should not change anything (
as long as we don't have nested signal handlers).
Out of curiosity, I also remove the ones in handlers (and keep the one in handle_sig_alarm()
at the top), and everything seems to work fine:
https://cirrus-ci.com/build/6277169619402752
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 12:42 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 14:37 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-02-03 06:19 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-02-03 17:09 ` Andres Freund <andres@anarazel.de>
2026-02-03 17:58 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 71+ messages in thread
From: Andres Freund @ 2026-02-03 17:09 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Álvaro Herrera <alvherre@kurilemu.de>; Fujii Masao <masao.fujii@gmail.com>; Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On 2026-02-03 06:19:13 +0000, Bertrand Drouvot wrote:
> On Fri, Jan 30, 2026 at 03:37:57PM +0100, Álvaro Herrera wrote:
> > (Though to be honest, it's not clear to me why it would matter at which
> > point in handle_sig_alarm we call SetLatch relative to the variables
> > being set, given that these variables are only going to matter once the
> > signal handler returns to the original code and the next
> > CHECK_FOR_INTERRUPTS is hit.)
>
> Yeah, I think that we could keep the SetLatch() at the top of handle_sig_alarm().
Why at the top, rather than at the bottom? I don't think / I hope today's
signal handlers rely on it, but in the past we had cases where some signal
handlers ran code that could lead to a ResetLatch() being done.
Why does it matter for your patch whether SetLatch() is done multiple times as
part of various timeout handlers? I don't see how repeated SetLatch() calls
could trigger more interference with ProcSleep()? Once the latch is set it is
set (and indeed SetLatch() just returns immediately if it already is set).
> My understanding is that the signal handler runs to completion without being interrupted
> by the code it interrupted.
Right. But they can, on some platforms at least, be interrupted by *other*
signal handlers. I don't see any reason to believe that is not happening at
the moment.
> Out of curiosity, I also remove the ones in handlers (and keep the one in handle_sig_alarm()
> at the top), and everything seems to work fine:
>
> https://cirrus-ci.com/build/6277169619402752
That doesn't tell you very much, I think. Our coverage of the relevant edge
cases isn't that good, I think.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <masao.fujii@gmail.com>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 12:42 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 14:37 ` Re: Flush some statistics within running transactions Álvaro Herrera <alvherre@kurilemu.de>
2026-02-03 06:19 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-03 17:09 ` Re: Flush some statistics within running transactions Andres Freund <andres@anarazel.de>
@ 2026-02-03 17:58 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-02-03 17:58 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Álvaro Herrera <alvherre@kurilemu.de>; Fujii Masao <masao.fujii@gmail.com>; Sami Imseih <samimseih@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Tue, Feb 03, 2026 at 12:09:31PM -0500, Andres Freund wrote:
> Hi,
>
> On 2026-02-03 06:19:13 +0000, Bertrand Drouvot wrote:
> > On Fri, Jan 30, 2026 at 03:37:57PM +0100, Álvaro Herrera wrote:
> > > (Though to be honest, it's not clear to me why it would matter at which
> > > point in handle_sig_alarm we call SetLatch relative to the variables
> > > being set, given that these variables are only going to matter once the
> > > signal handler returns to the original code and the next
> > > CHECK_FOR_INTERRUPTS is hit.)
> >
> Why does it matter for your patch whether SetLatch() is done multiple times as
> part of various timeout handlers? I don't see how repeated SetLatch() calls
> could trigger more interference with ProcSleep()? Once the latch is set it is
> set (and indeed SetLatch() just returns immediately if it already is set).
>
Yeah, this was just a finding while diagnosing the ProcSleep() "issue". This
discussion is not relevant in this thread anymore (specially since v5 where the
design changed in such a way that the ProcSleep() "issue" does not appear
anymore).
We could open a dedicated thread if we think that's worth continuing the discussion
about removing the SetLatch() in those handlers (but they are probably harmless
to keep afterall). Thanks to you and Álvaro for having shared your thoughts on
it.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
@ 2026-01-22 02:28 ` Michael Paquier <michael@paquier.xyz>
2026-01-22 08:02 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 71+ messages in thread
From: Michael Paquier @ 2026-01-22 02:28 UTC (permalink / raw)
To: Sami Imseih <samimseih@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
On Wed, Jan 21, 2026 at 07:41:30PM -0600, Sami Imseih wrote:
> Another one would be n_mod_since_analyze, That should
> only be updated after commit (or not after rollback). Otherwise,
> it may throw autovanalyze threshold calculations way off. Same
> for n_dead_tup and autovacuum.
Point taken. It sounds like it is going to be super important to
document in the patch these kind of current expectations, so as one
does not flip the flush mode one way or another incorrectly, or
assigns an incorrect flush mode when adding a new stats kind. It's
probably worth documenting that the end-of-transaction flush should be
the default norm, while the out-of-transaction case should be an
exception one needs to be careful of.
> Sure, Bertrand mentioned early in the thread that the anytime flushes
> could be made configurable. Perhaps that is a good idea where we can
> default with something large like 10s intervals for anytime flushes, but allow
> the user to configure a more frequent flushes ( although I would think
> that 1 sec is the minimum we should allow ).
Sure, I am just mentioning that we should not be that aggressive for
everybody. If this can be made configurable on a call-basis, even if
it means a new GUC, that may be better in the long run.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aXGLNhFVERJz-kl0@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-22 02:28 ` Re: Flush some statistics within running transactions Michael Paquier <michael@paquier.xyz>
@ 2026-01-22 08:02 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 71+ messages in thread
From: Bertrand Drouvot @ 2026-01-22 08:02 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Sami Imseih <samimseih@gmail.com>; pgsql-hackers@lists.postgresql.org, Zsolt Parragi <zsolt.parragi@percona.com>
Hi,
On Thu, Jan 22, 2026 at 11:28:06AM +0900, Michael Paquier wrote:
> On Wed, Jan 21, 2026 at 07:41:30PM -0600, Sami Imseih wrote:
> > Another one would be n_mod_since_analyze, That should
> > only be updated after commit (or not after rollback). Otherwise,
> > it may throw autovanalyze threshold calculations way off. Same
> > for n_dead_tup and autovacuum.
>
> Point taken. It sounds like it is going to be super important to
> document in the patch these kind of current expectations, so as one
> does not flip the flush mode one way or another incorrectly, or
> assigns an incorrect flush mode when adding a new stats kind. It's
> probably worth documenting that the end-of-transaction flush should be
> the default norm, while the out-of-transaction case should be an
> exception one needs to be careful of.
Agreed, I'll add more explanations around that.
> > Sure, Bertrand mentioned early in the thread that the anytime flushes
> > could be made configurable. Perhaps that is a good idea where we can
> > default with something large like 10s intervals for anytime flushes, but allow
> > the user to configure a more frequent flushes ( although I would think
> > that 1 sec is the minimum we should allow ).
>
> Sure, I am just mentioning that we should not be that aggressive for
> everybody.
I'm not opposed to increase the flush frequency but I suppose most of the monitoring
tools are sampling at a 1s frequency. So, if we set the flush frequency to say 10s,
that would result in "spikes" every 10s. That's misleading, because it's not a
spike in activity, it's a delay in reporting.
I think that would make sense if we expect the 1s interval to have a negative
impact, but that's not what I expect and observed.
> If this can be made configurable on a call-basis, even if
> it means a new GUC, that may be better in the long run.
If we think that the 1s interval is a problem, we could go in that direction.
Though it might be better to hardcode a larger value instead of letting the users
set values that could be problematic.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 71+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Re: Flush some statistics within running transactions Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Re: Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-01-20 23:18 ` Zsolt Parragi <zsolt.parragi@percona.com>
1 sibling, 0 replies; 71+ messages in thread
From: Zsolt Parragi @ 2026-01-20 23:18 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Sami Imseih <samimseih@gmail.com>; pgsql-hackers@lists.postgresql.org
Hello
@@ -264,6 +266,12 @@ typedef struct PgStat_KindInfo
/* Flush behavior */
PgStat_FlushBehavior flush_behavior;
+ /*
+ * For PGSTAT_FLUSH_MIXED kinds: callback to flush only some fields. If
+ * NULL for a MIXED kind, treated as PGSTAT_FLUSH_AT_TXN_BOUNDARY.
+ */
+ bool (*flush_anytime_cb) (PgStat_EntryRef *entry_ref, bool nowait);
+
The comment seems to use incorrect names, shouldn't be FLUSH_MIXED and
FLUSH_AT_TXN_BOUNDARY without PGSTAT_?
^ permalink raw reply [nested|flat] 71+ messages in thread
end of thread, other threads:[~2026-03-25 03:16 UTC | newest]
Thread overview: 71+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-01-12 11:03 Flush some statistics within running transactions Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 03:54 ` Sami Imseih <samimseih@gmail.com>
2026-01-15 12:18 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-15 17:25 ` Sami Imseih <samimseih@gmail.com>
2026-01-15 18:16 ` Sami Imseih <samimseih@gmail.com>
2026-01-16 11:37 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-16 16:44 ` Sami Imseih <samimseih@gmail.com>
2026-01-19 12:17 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 19:27 ` Sami Imseih <samimseih@gmail.com>
2026-01-21 10:34 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-21 23:41 ` Sami Imseih <samimseih@gmail.com>
2026-01-22 09:52 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 17:23 ` Sami Imseih <samimseih@gmail.com>
2026-01-26 07:35 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-27 16:53 ` Sami Imseih <samimseih@gmail.com>
2026-01-22 00:02 ` Michael Paquier <michael@paquier.xyz>
2026-01-22 01:41 ` Sami Imseih <samimseih@gmail.com>
2026-01-22 01:56 ` Fujii Masao <masao.fujii@gmail.com>
2026-01-22 07:43 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 12:12 ` Fujii Masao <masao.fujii@gmail.com>
2026-01-22 16:45 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-26 06:59 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-28 06:35 ` Michael Paquier <michael@paquier.xyz>
2026-01-28 11:14 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-31 01:33 ` Sami Imseih <samimseih@gmail.com>
2026-01-31 17:16 ` Sami Imseih <samimseih@gmail.com>
2026-02-02 07:31 ` Michael Paquier <michael@paquier.xyz>
2026-02-02 17:19 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-02 17:16 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-04 15:57 ` Zsolt Parragi <zsolt.parragi@percona.com>
2026-02-04 20:15 ` Sami Imseih <samimseih@gmail.com>
2026-02-04 16:19 ` Sami Imseih <samimseih@gmail.com>
2026-02-04 16:26 ` Sami Imseih <samimseih@gmail.com>
2026-02-06 11:09 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-14 02:23 ` Sami Imseih <samimseih@gmail.com>
2026-02-16 08:27 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-16 22:42 ` Sami Imseih <samimseih@gmail.com>
2026-02-17 07:04 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-17 19:18 ` Sami Imseih <samimseih@gmail.com>
2026-02-18 05:40 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-18 11:37 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2026-02-19 07:06 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 03:58 ` Michael Paquier <michael@paquier.xyz>
2026-02-19 08:01 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-19 22:08 ` Sami Imseih <samimseih@gmail.com>
2026-02-20 15:55 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 02:12 ` Sami Imseih <samimseih@gmail.com>
2026-02-23 08:14 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-23 23:47 ` Sami Imseih <samimseih@gmail.com>
2026-02-24 01:56 ` Michael Paquier <michael@paquier.xyz>
2026-02-24 12:01 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 06:26 ` Michael Paquier <michael@paquier.xyz>
2026-03-16 09:20 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-03-16 10:22 ` Michael Paquier <michael@paquier.xyz>
2026-03-16 23:42 ` Sami Imseih <samimseih@gmail.com>
2026-03-25 03:16 ` Michael Paquier <michael@paquier.xyz>
2026-02-23 23:48 ` Michael Paquier <michael@paquier.xyz>
2026-02-24 13:55 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-24 16:32 ` Sami Imseih <samimseih@gmail.com>
2026-02-26 03:33 ` Michael Paquier <michael@paquier.xyz>
2026-01-30 09:46 ` Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 12:42 ` Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 14:37 ` Álvaro Herrera <alvherre@kurilemu.de>
2026-01-30 15:10 ` Andres Freund <andres@anarazel.de>
2026-01-30 16:18 ` Álvaro Herrera <alvherre@kurilemu.de>
2026-02-03 06:19 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-02-03 17:09 ` Andres Freund <andres@anarazel.de>
2026-02-03 17:58 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-22 02:28 ` Michael Paquier <michael@paquier.xyz>
2026-01-22 08:02 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-01-20 23:18 ` Zsolt Parragi <zsolt.parragi@percona.com>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox