agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedRedesign per-backend statistics
11+ messages / 3 participants
[nested] [flat]
* Redesign per-backend statistics
@ 2026-08-03 14:20 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 11+ messages in thread
From: Bertrand Drouvot @ 2026-08-03 14:20 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>
Hi hackers,
The current per-backend statistics design excludes several process types and
duplicates accounting and storage: WAL, lock, and IO activity is reported both to
the global statistics and to PGSTAT_KIND_BACKEND.
This duplicated work was raised as a concern in [1], and its cost would grow as
more statistics acquire per-backend reporting.
Andres proposed a new design in [2]:
"
I think what we'd want is rather something where for each applicable stats kind
we have a shared counter for all exited backends and then per-backend counters
for live backends, with helpers to aggregate the exited + live stats to a total
"
and this is what the attached patch series is implementing.
Its main benefits are:
1/ each counter is reported and stored once, rather than in both the fixed global
statistics and PGSTAT_KIND_BACKEND.
2/ WAL, lock, and IO flushes update a per backend entry without acquiring
the corresponding global statistics lock or performing a hash lookup. This removes
global lock contention from the flush path.
3/ WAL, lock, and IO statistics have independent storage and locking instead of
sharing one PGSTAT_KIND_BACKEND entry.
4/ per-backend statistics become available for auxiliary processes previously
excluded by the backend type filtering.
There are 2 tradeoff though:
1/ fetching global statistics now requires combining all live entries.
2/ a shared statistics reset now clears both the global stats and all live entries.
Otherwise, values from live entries would immediately reappear in the global view.
The first tradeoff moves work from the frequent flush path to the comparatively
infrequent query path. The second is a documented behavior change.
The patch series is organized that way:
0001: add tests for per-backend statistics
It adds new tests that will serve as compatibility coverage for the redesign.
It could be applied while we are discussing the other patches.
0002: add new infrastructure for per-backend statistics
It introduces the common per-backend entry header, the kind metadata needed to
describe per-backend storage, and backend-local state holding the attached dshash
and cached entry pointer. It provides the common operations for creating and
attaching the hashes, creating and caching the current process's entries, fetching
an individual entry, constructing consistent snapshots, transferring entries into
global statistics, removing entries, and the accumulation at clean server shutdown.
No statistics kind registers per-backend metadata in this patch.
0003 moves WAL statistics to the infrastructure introduced by 0002.
0004 performs the corresponding conversion for lock statistics.
0005 performs the corresponding conversion for IO statistics. Once WAL, lock,
and IO have moved, PGSTAT_KIND_BACKEND contains no data, so this patch also removes
that kind and pgstat_backend.c, together with their obsolete infrastructure.
Design explanation for the new hashes:
- a fixed array indexed by ProcNumber would avoid hash operations, but it would
reserve shared memory for every possible process slot and every participating kind.
Queries could also have to inspect unused slots. The dshash allocates entries
for processes that actually exist and lets queries iterate those entries directly.
- reusing the current variable numbered statistics hash would require aggregate
queries to scan unrelated statistics entries or require another structure for
enumerating only the live entries of that kind. A dedicated per kind hash provides
that enumeration directly.
Remark:
The patch series limits this new infrastructure to built in fixed numbered
statistics kinds as this is the only use case we have had so far. We could extend
to variable ones later on if needed.
[1]: https://postgr.es/m/7fhpds4xqk6bnudzmzkqi33pinsxammpljwde5gfkjdygvejrj@ojkzfr7dxkmm
[2]: https://postgr.es/m/et272fdhdx6yphlgzvrgsf7bgwnf3vqciwp4gxqubro42uaflp%40ohslaocvwgvi
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v1-0001-pgstat-add-tests-for-per-backend-statistics.patch (21.1K, ../../anCjoFkGah%2FuB7lw@bdtpg/2-v1-0001-pgstat-add-tests-for-per-backend-statistics.patch)
download | inline diff:
From abc8330baf690a036114ea688fcaa4643ab641c9 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:23:40 +0000
Subject: [PATCH v1 1/5] pgstat: add tests for per-backend statistics
Add an isolation test for per-backend WAL statistics. Verify that SNAPSHOT, CACHE
and NONE modes behave as expected.
Add a TAP test that starts a backend after another session has built a
statistics snapshot. Per-backend WAL, IO, and Lock queries for the new
backend must return no rows, rather than fabricated rows containing zeroes.
Extend test_shm_mq with a shared-memory-only worker that reports WAL usage
through a routine nonblocking flush and remains alive while the controlling
backend verifies that the global WAL counters include it.
Finally, extend the core statistics regression test for per-backend WAL
reset. Check that resetting one backend reduces its counters and sets its reset
timestamp without removing the same activity from the global WAL statistics.
These tests will serve as compatibility coverage for the per-backend redesign
changes coming in following commits.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by:
Discussion:
---
.../isolation/expected/stats-per-backend.out | 143 ++++++++++++++++++
src/test/isolation/isolation_schedule | 1 +
.../isolation/specs/stats-per-backend.spec | 123 +++++++++++++++
src/test/modules/test_misc/meson.build | 1 +
.../modules/test_misc/t/015_stats_snapshot.pl | 48 ++++++
.../test_shm_mq/expected/test_shm_mq.out | 7 +
.../modules/test_shm_mq/sql/test_shm_mq.sql | 3 +
src/test/modules/test_shm_mq/test.c | 51 +++++++
.../modules/test_shm_mq/test_shm_mq--1.0.sql | 4 +
src/test/modules/test_shm_mq/test_shm_mq.h | 4 +
src/test/modules/test_shm_mq/worker.c | 10 ++
src/test/regress/expected/stats.out | 29 ++++
src/test/regress/sql/stats.sql | 10 ++
13 files changed, 434 insertions(+)
30.6% src/test/isolation/expected/
31.4% src/test/isolation/specs/
9.7% src/test/modules/test_misc/t/
17.6% src/test/modules/test_shm_mq/
5.6% src/test/regress/expected/
4.3% src/test/regress/sql/
diff --git a/src/test/isolation/expected/stats-per-backend.out b/src/test/isolation/expected/stats-per-backend.out
new file mode 100644
index 00000000000..0d85ae99033
--- /dev/null
+++ b/src/test/isolation/expected/stats-per-backend.out
@@ -0,0 +1,143 @@
+Parsed test spec with 2 sessions
+
+starting permutation: s1_fetch_consistency_snapshot s1_begin s1_build_snapshot s2_generate s1_check_snapshot_backend s1_commit s1_check_live_backend
+step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot';
+step s1_begin: BEGIN;
+step s1_build_snapshot:
+ SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal;
+
+snapshot_created
+----------------
+t
+(1 row)
+
+step s2_generate:
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_check_snapshot_backend:
+ SELECT wal_records = 0 AS snapshot_excludes_later_update
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+snapshot_excludes_later_update
+------------------------------
+t
+(1 row)
+
+step s1_commit: COMMIT;
+step s1_check_live_backend:
+ SELECT wal_records > 0 AS live_entry_exists
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+live_entry_exists
+-----------------
+t
+(1 row)
+
+
+starting permutation: s1_fetch_consistency_cache s2_generate s1_begin s1_save_backend_stats s2_generate_more s1_check_cached_backend s1_commit s1_check_refreshed_backend
+step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache';
+step s2_generate:
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_begin: BEGIN;
+step s1_save_backend_stats:
+ INSERT INTO stats_per_backend_saved
+ SELECT wal_records
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+step s2_generate_more:
+ INSERT INTO stats_per_backend_data VALUES (2);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_check_cached_backend:
+ SELECT current.wal_records = saved.wal_records AS cache_is_stable
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+
+cache_is_stable
+---------------
+t
+(1 row)
+
+step s1_commit: COMMIT;
+step s1_check_refreshed_backend:
+ SELECT current.wal_records > saved.wal_records AS cache_is_refreshed
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+
+cache_is_refreshed
+------------------
+t
+(1 row)
+
+
+starting permutation: s1_reset_backend s1_check_backend_reset
+step s1_reset_backend:
+ SELECT pg_stat_reset_backend_stats(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+pg_stat_reset_backend_stats
+---------------------------
+
+(1 row)
+
+step s1_check_backend_reset:
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+reset_timestamp_set
+-------------------
+t
+(1 row)
+
+
+starting permutation: s1_reset_shared s1_check_shared_reset
+step s1_reset_shared:
+ SELECT pg_stat_reset_shared('wal');
+
+pg_stat_reset_shared
+--------------------
+
+(1 row)
+
+step s1_check_shared_reset:
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+reset_timestamp_set
+-------------------
+t
+(1 row)
+
diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule
index 26abed9f9f0..5d97c56b5d6 100644
--- a/src/test/isolation/isolation_schedule
+++ b/src/test/isolation/isolation_schedule
@@ -104,6 +104,7 @@ test: vacuum-concurrent-drop
test: vacuum-conflict
test: vacuum-skip-locked
test: stats
+test: stats-per-backend
test: horizons
test: predicate-hash
test: predicate-gist
diff --git a/src/test/isolation/specs/stats-per-backend.spec b/src/test/isolation/specs/stats-per-backend.spec
new file mode 100644
index 00000000000..5dfc8dbe90e
--- /dev/null
+++ b/src/test/isolation/specs/stats-per-backend.spec
@@ -0,0 +1,123 @@
+# Test snapshot, cache, and reset behavior for per-backend statistics.
+#
+# Session s2 generates and flushes WAL records, which provide a deterministic
+# per-backend counter. Session s1 verifies that SNAPSHOT mode fixes all
+# per-backend entries at the initial statistics snapshot, while CACHE mode
+# fixes each entry on first access and refreshes it after the transaction
+# ends. The final permutations verify reset timestamps after per-backend and
+# shared resets.
+
+setup
+{
+ CREATE TABLE stats_per_backend_data(id int);
+ CREATE TABLE stats_per_backend_saved(wal_records bigint);
+}
+
+teardown
+{
+ DROP TABLE stats_per_backend_data;
+ DROP TABLE stats_per_backend_saved;
+}
+
+session s1
+setup { SET stats_fetch_consistency = 'none'; }
+
+step s1_fetch_consistency_cache { SET stats_fetch_consistency = 'cache'; }
+step s1_fetch_consistency_snapshot { SET stats_fetch_consistency = 'snapshot'; }
+step s1_begin { BEGIN; }
+step s1_commit { COMMIT; }
+step s1_build_snapshot {
+ SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal;
+}
+step s1_check_snapshot_backend {
+ SELECT wal_records = 0 AS snapshot_excludes_later_update
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_live_backend {
+ SELECT wal_records > 0 AS live_entry_exists
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_save_backend_stats {
+ INSERT INTO stats_per_backend_saved
+ SELECT wal_records
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_cached_backend {
+ SELECT current.wal_records = saved.wal_records AS cache_is_stable
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+}
+step s1_check_refreshed_backend {
+ SELECT current.wal_records > saved.wal_records AS cache_is_refreshed
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+}
+step s1_reset_backend {
+ SELECT pg_stat_reset_backend_stats(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_backend_reset {
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_reset_shared {
+ SELECT pg_stat_reset_shared('wal');
+}
+step s1_check_shared_reset {
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+
+session s2
+step s2_generate {
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+}
+step s2_generate_more {
+ INSERT INTO stats_per_backend_data VALUES (2);
+ SELECT pg_stat_force_next_flush();
+}
+
+# with stats_fetch_consistency=snapshot s1 should not see flushed changes from
+# s2 after building the statistics snapshot, but should see them after commit
+permutation
+ s1_fetch_consistency_snapshot
+ s1_begin
+ s1_build_snapshot
+ s2_generate
+ s1_check_snapshot_backend
+ s1_commit
+ s1_check_live_backend
+
+# with stats_fetch_consistency=cache s1 should not see flushed changes from s2
+# after the first access, but should see them after commit
+permutation
+ s1_fetch_consistency_cache
+ s2_generate
+ s1_begin
+ s1_save_backend_stats
+ s2_generate_more
+ s1_check_cached_backend
+ s1_commit
+ s1_check_refreshed_backend
+
+# a per-backend reset should set that backend's reset timestamp
+permutation s1_reset_backend s1_check_backend_reset
+
+# a shared WAL reset should set the reset timestamp for live backends
+permutation s1_reset_shared s1_check_shared_reset
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index ee290698b31..cf4c8558f3d 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -23,6 +23,7 @@ tests += {
't/012_ddlutils.pl',
't/013_temp_obj_multisession.pl',
't/014_log_statement_max_length.pl',
+ 't/015_stats_snapshot.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/015_stats_snapshot.pl b/src/test/modules/test_misc/t/015_stats_snapshot.pl
new file mode 100644
index 00000000000..43c9157bbb4
--- /dev/null
+++ b/src/test/modules/test_misc/t/015_stats_snapshot.pl
@@ -0,0 +1,48 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Verify that a statistics snapshot excludes backends that started after the
+# snapshot was created.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('stats_snapshot');
+$node->init;
+$node->start;
+
+my $snapshot = $node->background_psql('postgres');
+
+$snapshot->query_safe(
+ q[
+ BEGIN;
+ SET LOCAL stats_fetch_consistency = 'snapshot';
+ SELECT count(*) FROM pg_stat_wal;
+]);
+
+# Establish this connection only after the first session built its snapshot.
+my $late = $node->background_psql('postgres');
+my $late_pid = $late->query_safe('SELECT pg_backend_pid()');
+
+# WAL returns one scalar composite, so count a field to distinguish a NULL
+# result from the fabricated all-zero row.
+is( $snapshot->query_safe(
+ qq[
+ SELECT
+ (SELECT count(wal_records)
+ FROM pg_stat_get_backend_wal($late_pid)),
+ (SELECT count(*) FROM pg_stat_get_backend_io($late_pid)),
+ (SELECT count(*) FROM pg_stat_get_backend_lock($late_pid));
+ ]),
+ '0|0|0',
+ 'statistics snapshot excludes a backend created later');
+
+$snapshot->query_safe('ROLLBACK');
+$late->quit;
+$snapshot->quit;
+$node->stop;
+
+done_testing();
diff --git a/src/test/modules/test_shm_mq/expected/test_shm_mq.out b/src/test/modules/test_shm_mq/expected/test_shm_mq.out
index c4858b0c205..ec3a7d3199e 100644
--- a/src/test/modules/test_shm_mq/expected/test_shm_mq.out
+++ b/src/test/modules/test_shm_mq/expected/test_shm_mq.out
@@ -34,3 +34,10 @@ SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int
(1 row)
+-- A shmem-only worker can publish statistics before its clean exit.
+SELECT test_shm_mq_worker_stats();
+ test_shm_mq_worker_stats
+--------------------------
+ t
+(1 row)
+
diff --git a/src/test/modules/test_shm_mq/sql/test_shm_mq.sql b/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
index 9de19d304a2..e80e4399881 100644
--- a/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
+++ b/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
@@ -10,3 +10,6 @@ SELECT test_shm_mq(1024, 'a', 2001, 1);
SELECT test_shm_mq(32768, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,(100+900*random())::int)), 10000, 1);
SELECT test_shm_mq(100, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,(100+200*random())::int)), 10000, 1);
SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,270000)), 200, 3);
+
+-- A shmem-only worker can publish statistics before its clean exit.
+SELECT test_shm_mq_worker_stats();
diff --git a/src/test/modules/test_shm_mq/test.c b/src/test/modules/test_shm_mq/test.c
index 0e55287e510..4464d796ef9 100644
--- a/src/test/modules/test_shm_mq/test.c
+++ b/src/test/modules/test_shm_mq/test.c
@@ -26,6 +26,7 @@ PG_MODULE_MAGIC;
PG_FUNCTION_INFO_V1(test_shm_mq);
PG_FUNCTION_INFO_V1(test_shm_mq_pipelined);
+PG_FUNCTION_INFO_V1(test_shm_mq_worker_stats);
static void verify_message(Size origlen, char *origdata, Size newlen,
char *newdata);
@@ -253,6 +254,56 @@ test_shm_mq_pipelined(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+/*
+ * Verify that a shmem-only worker can publish statistics through a routine
+ * nonblocking flush while it remains alive.
+ */
+Datum
+test_shm_mq_worker_stats(PG_FUNCTION_ARGS)
+{
+ const char *message = PG_TEST_SHM_MQ_STATS_MESSAGE;
+ PgStat_Counter before;
+ PgStat_Counter after;
+ dsm_segment *seg;
+ shm_mq_handle *outqh;
+ shm_mq_handle *inqh;
+ shm_mq_result res;
+ Size len;
+ void *data;
+
+ pgstat_clear_snapshot();
+ before = pgstat_fetch_stat_wal()->wal_counters.wal_records;
+
+ test_shm_mq_setup(1024, 1, &seg, &outqh, &inqh);
+
+ res = shm_mq_send(outqh, strlen(message), message, false, true);
+
+ if (res != SHM_MQ_SUCCESS)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not send statistics test message")));
+
+ res = shm_mq_receive(inqh, &len, &data, false);
+
+ if (res != SHM_MQ_SUCCESS)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not receive statistics test message")));
+
+ verify_message(strlen(message), (char *) message, len, data);
+
+ /*
+ * The worker is waiting for another message here. Check its live entry
+ * before detaching the queues and triggering its shutdown flush.
+ */
+ pgstat_clear_snapshot();
+ after = pgstat_fetch_stat_wal()->wal_counters.wal_records;
+
+ dsm_detach(seg);
+
+ PG_RETURN_BOOL(after >= before + PG_TEST_SHM_MQ_STATS_RECORDS);
+}
+
/*
* Verify that two messages are the same.
*/
diff --git a/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql b/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
index 56db05d93df..dce6f7378a5 100644
--- a/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
+++ b/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
@@ -17,3 +17,7 @@ CREATE FUNCTION test_shm_mq_pipelined(queue_size pg_catalog.int8,
verify pg_catalog.bool default true)
RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION test_shm_mq_worker_stats()
+ RETURNS pg_catalog.bool
+ AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_shm_mq/test_shm_mq.h b/src/test/modules/test_shm_mq/test_shm_mq.h
index b6a0290289c..501be180642 100644
--- a/src/test/modules/test_shm_mq/test_shm_mq.h
+++ b/src/test/modules/test_shm_mq/test_shm_mq.h
@@ -21,6 +21,10 @@
/* Identifier for shared memory segments used by this extension. */
#define PG_TEST_SHM_MQ_MAGIC 0x79fb2447
+/* Special message used to test statistics from a shmem-only worker. */
+#define PG_TEST_SHM_MQ_STATS_MESSAGE "test_shm_mq worker stats"
+#define PG_TEST_SHM_MQ_STATS_RECORDS 1000000
+
/*
* This structure is stored in the dynamic shared memory segment. We use
* it to determine whether all workers started up OK and successfully
diff --git a/src/test/modules/test_shm_mq/worker.c b/src/test/modules/test_shm_mq/worker.c
index 0ba1cfcac47..afcea7fc0cb 100644
--- a/src/test/modules/test_shm_mq/worker.c
+++ b/src/test/modules/test_shm_mq/worker.c
@@ -19,6 +19,7 @@
#include "postgres.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -27,6 +28,7 @@
#include "storage/shm_mq.h"
#include "storage/shm_toc.h"
#include "tcop/tcopprot.h"
+#include "utils/pgstat_internal.h"
#include "test_shm_mq.h"
@@ -185,6 +187,14 @@ copy_messages(shm_mq_handle *inqh, shm_mq_handle *outqh)
if (res != SHM_MQ_SUCCESS)
break;
+ if (len == strlen(PG_TEST_SHM_MQ_STATS_MESSAGE) &&
+ memcmp(data, PG_TEST_SHM_MQ_STATS_MESSAGE, len) == 0)
+ {
+ pgWalUsage.wal_records += PG_TEST_SHM_MQ_STATS_RECORDS;
+ pgstat_report_fixed = true;
+ (void) pgstat_report_stat(false);
+ }
+
/* Send it back out. */
res = shm_mq_send(outqh, len, data, false, true);
if (res != SHM_MQ_SUCCESS)
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index e230356de13..fae5f939523 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1044,6 +1044,35 @@ SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_bac
t
(1 row)
+-- Test pg_stat_reset_backend_stats() for WAL
+SELECT wal_bytes AS wal_bytes_before_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT pg_stat_reset_backend_stats(pg_backend_pid());
+ pg_stat_reset_backend_stats
+-----------------------------
+
+(1 row)
+
+SELECT wal_bytes AS wal_bytes_after_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT :wal_bytes_after_reset < :wal_bytes_before_reset;
+ ?column?
+----------
+ t
+(1 row)
+
+-- global view should still have the data
+SELECT wal_bytes >= :wal_bytes_before_reset FROM pg_stat_wal;
+ ?column?
+----------
+ t
+(1 row)
+
+-- reset timestamp should be set
+SELECT stats_reset IS NOT NULL FROM pg_stat_get_backend_wal(pg_backend_pid());
+ ?column?
+----------
+ t
+(1 row)
+
-- Test pg_stat_get_backend_idset() and some allied functions.
-- In particular, verify that their notion of backend ID matches
-- our temp schema index.
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 4c265d1245c..c17753dcd20 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -482,6 +482,16 @@ SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal;
SELECT pg_stat_force_next_flush();
SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_backend_pid());
+-- Test pg_stat_reset_backend_stats() for WAL
+SELECT wal_bytes AS wal_bytes_before_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT pg_stat_reset_backend_stats(pg_backend_pid());
+SELECT wal_bytes AS wal_bytes_after_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT :wal_bytes_after_reset < :wal_bytes_before_reset;
+-- global view should still have the data
+SELECT wal_bytes >= :wal_bytes_before_reset FROM pg_stat_wal;
+-- reset timestamp should be set
+SELECT stats_reset IS NOT NULL FROM pg_stat_get_backend_wal(pg_backend_pid());
+
-- Test pg_stat_get_backend_idset() and some allied functions.
-- In particular, verify that their notion of backend ID matches
-- our temp schema index.
--
2.34.1
[text/x-diff] v1-0002-pgstat-add-new-infrastructure-for-per-backend-sta.patch (23.1K, ../../anCjoFkGah%2FuB7lw@bdtpg/3-v1-0002-pgstat-add-new-infrastructure-for-per-backend-sta.patch)
download | inline diff:
From b1d28f72cfc6fe8925a334a0d8f7da0626d1cf9d Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 29 Jul 2026 14:11:27 +0000
Subject: [PATCH v1 2/5] pgstat: add new infrastructure for per-backend
statistics
Add a new infrastructure for per-backend statistics that keep one shared entry
per live backend while keeping their existing shared data as global stats for
exited backends.
Define a common dshash entry header containing the ProcNumber key, BackendType,
and content LWLock. Extend PgStat_KindInfo with per-backend statistics related
informations.
At shared-memory initialization, create a dshash for every participating kind.
Create entries during pgstat_initialize(), so that auxiliary and shared memory
only processes are covered. Nonblocking flushes can then acquire the cached
entry's content lock without performing a dshash lookup, DSA address resolution,
allocation, or hash resize.
Add helpers to fetch per-backend entries, include live entries in global
snapshots, and transfer entries to the global statistics when backends exit.
Add a local cache keyed by statistics kind and ProcNumber for per-backend
fetches.
No statistics kind registers per backend metadata in this commit, so the new
shared memory creation and backend initialization loops are no ops. Subsequent
commits will add WAL, Lock, and IO statistics into the infrastructure individually.
Limit this infrastructure to built in fixed numbered statistics kinds as this is
the only use case we have had so far. We could extend to variable ones later on
if needed.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by:
Discussion:
---
src/backend/utils/activity/pgstat.c | 506 ++++++++++++++++++++++
src/backend/utils/activity/pgstat_shmem.c | 46 ++
src/include/utils/pgstat_internal.h | 43 ++
src/tools/pgindent/typedefs.list | 4 +
4 files changed, 599 insertions(+)
89.6% src/backend/utils/activity/
9.6% src/include/utils/
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 50cd07822b4..efc7d427f22 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -175,6 +175,41 @@ typedef struct PgStat_SnapshotEntry
#define SH_DECLARE
#include "lib/simplehash.h"
+/* hash table for per-backend stats snapshot entries */
+typedef struct PgStat_PerBackendSnapshotKey
+{
+ PgStat_Kind kind;
+ ProcNumber procnum;
+} PgStat_PerBackendSnapshotKey;
+
+typedef struct PgStat_PerBackendSnapshotEntry
+{
+ PgStat_PerBackendSnapshotKey key;
+ char status; /* for simplehash use */
+ void *data; /* the stats data itself */
+} PgStat_PerBackendSnapshotEntry;
+
+#define SH_PREFIX pgstat_per_backend_snapshot
+#define SH_ELEMENT_TYPE PgStat_PerBackendSnapshotEntry
+#define SH_KEY_TYPE PgStat_PerBackendSnapshotKey
+#define SH_KEY key
+#define SH_HASH_KEY(tb, key) \
+ fasthash32((const char *) &key, sizeof(PgStat_PerBackendSnapshotKey), 0)
+#define SH_EQUAL(tb, a, b) \
+ (memcmp(&a, &b, sizeof(PgStat_PerBackendSnapshotKey)) == 0)
+#define SH_SCOPE static inline
+#define SH_DEFINE
+#define SH_DECLARE
+#include "lib/simplehash.h"
+
+/* Per-kind, backend-local state for the per-backend dshashes. */
+typedef struct PgStat_PerBackendLocalState
+{
+ dshash_table *hash;
+ PgStatShared_PerBackendEntry *my_entry;
+} PgStat_PerBackendLocalState;
+
+static PgStat_PerBackendLocalState per_backend_states[PGSTAT_KIND_BUILTIN_SIZE];
/* ----------
* Local function forward declarations
@@ -195,6 +230,7 @@ static void pgstat_build_snapshot(void);
static void pgstat_build_snapshot_fixed(PgStat_Kind kind);
static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
+static void pgstat_create_my_per_backend_entries(void);
/* ----------
@@ -673,6 +709,13 @@ pgstat_initialize(void)
pgstat_attach_shmem();
+ /*
+ * Create and cache per-backend statistics entries here. This also covers
+ * processes that never call InitPostgres(), such as shared-memory-only
+ * background workers.
+ */
+ pgstat_create_my_per_backend_entries();
+
pgstat_init_snapshot_fixed();
/* Backend initialization callbacks */
@@ -946,6 +989,7 @@ pgstat_clear_snapshot(void)
/* Reset variables */
pgStatLocal.snapshot.context = NULL;
+ pgStatLocal.snapshot.per_backend_stats = NULL;
}
/*
@@ -1161,6 +1205,463 @@ pgstat_prep_snapshot(void)
NULL);
}
+/*
+ * Look up a per-backend stats entry in the backend snapshot hash.
+ *
+ * The returned entry may be empty when no matching statistics were found on
+ * first access.
+ */
+static PgStat_PerBackendSnapshotEntry *
+pgstat_lookup_per_backend_entry(PgStat_Kind kind, ProcNumber procnum)
+{
+ PgStat_PerBackendSnapshotKey key;
+
+ if (pgStatLocal.snapshot.per_backend_stats == NULL)
+ return NULL;
+
+ key.kind = kind;
+ key.procnum = procnum;
+
+ return pgstat_per_backend_snapshot_lookup(pgStatLocal.snapshot.per_backend_stats,
+ key);
+}
+
+/*
+ * Cache a per-backend stats entry in the backend snapshot hash.
+ *
+ * If data is NULL, cache an empty entry to record that no matching statistics
+ * were found on first access.
+ */
+static void *
+pgstat_cache_per_backend_entry(PgStat_Kind kind, ProcNumber procnum,
+ const void *data)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendSnapshotKey key;
+ PgStat_PerBackendSnapshotEntry *entry;
+ bool found;
+
+ Assert(pgstat_fetch_consistency > PGSTAT_FETCH_CONSISTENCY_NONE);
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ /* Ensure snapshot context exists */
+ if (!pgStatLocal.snapshot.context)
+ pgStatLocal.snapshot.context = AllocSetContextCreate(TopMemoryContext,
+ "PgStat Snapshot",
+ ALLOCSET_SMALL_SIZES);
+
+ /* Create per-backend hash on first use */
+ if (pgStatLocal.snapshot.per_backend_stats == NULL)
+ pgStatLocal.snapshot.per_backend_stats =
+ pgstat_per_backend_snapshot_create(pgStatLocal.snapshot.context, 64, NULL);
+
+ key.kind = kind;
+ key.procnum = procnum;
+
+ /* If already cached, return cached data */
+ entry = pgstat_per_backend_snapshot_lookup(pgStatLocal.snapshot.per_backend_stats, key);
+
+ if (entry)
+ return entry->data;
+
+ /* Insert new entry into the hash */
+ entry = pgstat_per_backend_snapshot_insert(pgStatLocal.snapshot.per_backend_stats,
+ key, &found);
+ Assert(!found);
+
+ if (data != NULL)
+ {
+ entry->data = MemoryContextAlloc(pgStatLocal.snapshot.context,
+ kind_info->per_backend_data_len);
+ memcpy(entry->data, data, kind_info->per_backend_data_len);
+ }
+ else
+ entry->data = NULL;
+
+ return entry->data;
+}
+
+static inline PgStat_PerBackendLocalState *
+pgstat_get_per_backend_local_state(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info PG_USED_FOR_ASSERTS_ONLY = pgstat_get_kind_info(kind);
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->fixed_amount);
+
+ if (!pgstat_is_kind_builtin(kind))
+ elog(ERROR, "invalid statistics kind: %u", kind);
+
+ return &per_backend_states[kind];
+}
+
+/*
+ * Create and cache this process's entry for one per-backend statistics kind.
+ */
+static PgStatShared_PerBackendEntry *
+pgstat_create_my_per_backend_entry(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ dshash_table *hash = pgstat_per_backend_attach(kind);
+ PgStatShared_PerBackendEntry *entry;
+ bool found;
+
+ Assert(state->my_entry == NULL);
+
+ if (hash == NULL)
+ return NULL;
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ entry = dshash_find_or_insert(hash, &MyProcNumber, &found);
+
+ /*
+ * A forced flush during early backend startup may already have created
+ * the entry. Preserve any statistics it contains.
+ */
+ if (!found)
+ {
+ entry->backend_type = MyBackendType;
+ LWLockInitialize(&entry->lock, LWTRANCHE_PGSTATS_DATA);
+ memset((char *) entry + kind_info->per_backend_data_off, 0,
+ kind_info->per_backend_data_len);
+ }
+
+ state->my_entry = entry;
+ dshash_release_lock(hash, entry);
+
+ return entry;
+}
+
+/*
+ * Create entries for all the kinds that use per-backend dshashes.
+ *
+ * Allocations and dshash resizes are deliberately done during backend
+ * initialization so routine stats flushes only need to conditionally acquire
+ * the cached entry's content lock.
+ */
+static void
+pgstat_create_my_per_backend_entries(void)
+{
+ for (PgStat_Kind kind = PGSTAT_KIND_BUILTIN_MIN;
+ kind <= PGSTAT_KIND_BUILTIN_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (kind_info == NULL || kind_info->per_backend_data_len == 0)
+ continue;
+
+ (void) pgstat_create_my_per_backend_entry(kind);
+ }
+}
+
+/*
+ * Attach to the per-backend dshash for the given kind.
+ * Returns NULL if the hash is not available (e.g. during bootstrap or
+ * if this kind doesn't have per-backend tracking).
+ */
+dshash_table *
+pgstat_per_backend_attach(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table_handle *handle_ptr;
+ MemoryContext oldcontext;
+ dshash_parameters params;
+
+ if (state->hash != NULL)
+ return state->hash;
+
+ if (!kind_info || !kind_info->per_backend_data_len)
+ return NULL;
+
+ /* Get the shared struct for this kind */
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ handle_ptr = (dshash_table_handle *) (shared_struct + kind_info->per_backend_hash_handle_off);
+
+ if (*handle_ptr == DSHASH_HANDLE_INVALID)
+ return NULL;
+
+ /*
+ * Build dshash parameters from kind info. All per-backend hashes use
+ * ProcNumber keys and dshash_memcmp/dshash_memhash.
+ */
+ params.key_size = sizeof(ProcNumber);
+ params.entry_size = kind_info->per_backend_data_off + kind_info->per_backend_data_len;
+ params.compare_function = dshash_memcmp;
+ params.hash_function = dshash_memhash;
+ params.copy_function = dshash_memcpy;
+ params.tranche_id = LWTRANCHE_PGSTATS_HASH;
+
+ /* Attach in TopMemoryContext */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+ state->hash = dshash_attach(pgStatLocal.dsa, ¶ms, *handle_ptr, NULL);
+ MemoryContextSwitchTo(oldcontext);
+
+ return state->hash;
+}
+
+/*
+ * Lock this process's cached entry for a per-backend statistics kind.
+ *
+ * A missing entry is only created in the blocking path. The routine nowait
+ * path must not perform dshash lookups, allocations, or DSA address
+ * resolution.
+ */
+void *
+pgstat_lock_my_per_backend_entry(PgStat_Kind kind, bool nowait)
+{
+ const PgStat_KindInfo *kind_info PG_USED_FOR_ASSERTS_ONLY = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ PgStatShared_PerBackendEntry *entry = state->my_entry;
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ if (entry == NULL)
+ {
+ if (nowait)
+ return NULL;
+
+ entry = pgstat_create_my_per_backend_entry(kind);
+ if (entry == NULL)
+ return NULL;
+ }
+
+ if (nowait)
+ {
+ if (!LWLockConditionalAcquire(&entry->lock, LW_EXCLUSIVE))
+ return NULL;
+ }
+ else
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ return entry;
+}
+
+/*
+ * Accumulate all live per-backend entries into the kind's data in
+ * pgStatLocal.snapshot, optionally caching each entry for SNAPSHOT mode.
+ */
+void
+pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash, void *snap)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ dshash_seq_status hstat;
+ PgStatShared_PerBackendEntry *entry;
+
+ dshash_seq_init(&hstat, hash, false);
+
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->lock, LW_SHARED);
+
+ /* Kind-specific accumulation into global snapshot */
+ kind_info->per_backend_acc_cb(snap, entry);
+
+ /*
+ * In SNAPSHOT mode, cache each per-backend entry so that
+ * pgstat_fetch_per_backend() can return a consistent point-in-time
+ * view without re-reading from shared memory.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ {
+ pgstat_cache_per_backend_entry(kind, entry->key,
+ (char *) entry + kind_info->per_backend_data_off);
+ }
+
+ LWLockRelease(&entry->lock);
+ }
+
+ dshash_seq_term(&hstat);
+}
+
+/*
+ * Fetch per-backend stats for the given kind and ProcNumber.
+ * Returns NULL if no entry exists. In NONE mode, returns a copy allocated in
+ * the current memory context. In CACHE and SNAPSHOT modes, returns a pointer
+ * owned by the statistics snapshot cache.
+ */
+void *
+pgstat_fetch_per_backend(PgStat_Kind kind, ProcNumber procnum)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ dshash_table *hash;
+ PgStatShared_PerBackendEntry *entry;
+ void *stats_data;
+ PgStat_PerBackendSnapshotEntry *snapshot_entry;
+
+ if (force_stats_snapshot_clear)
+ pgstat_clear_snapshot();
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return NULL;
+
+ /* In NONE mode, read directly and don't cache */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
+ {
+ entry = dshash_find(hash, &procnum, false);
+ if (entry == NULL)
+ return NULL;
+
+ LWLockAcquire(&entry->lock, LW_SHARED);
+ stats_data = palloc(kind_info->per_backend_data_len);
+ memcpy(stats_data, (char *) entry + kind_info->per_backend_data_off,
+ kind_info->per_backend_data_len);
+ LWLockRelease(&entry->lock);
+ dshash_release_lock(hash, entry);
+
+ return stats_data;
+ }
+
+ Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE ||
+ pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT);
+
+ /*
+ * Building a full snapshot pre-caches all existing per-backend entries.
+ * CACHE mode only needs the requested entry, so it must not build the
+ * aggregate fixed-kind snapshot and scan every live backend.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ pgstat_snapshot_fixed(kind);
+
+ snapshot_entry = pgstat_lookup_per_backend_entry(kind, procnum);
+
+ if (snapshot_entry != NULL)
+ return snapshot_entry->data;
+
+ /*
+ * Once a full snapshot has been built, a cache miss means the entry did
+ * not exist at the snapshot point. Do not admit an entry created later.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ return NULL;
+
+ /* CACHE miss: copy the live entry directly into the snapshot context. */
+ entry = dshash_find(hash, &procnum, false);
+
+ if (entry == NULL)
+ return pgstat_cache_per_backend_entry(kind, procnum, NULL);
+
+ LWLockAcquire(&entry->lock, LW_SHARED);
+
+ stats_data = pgstat_cache_per_backend_entry(kind, procnum,
+ (char *) entry + kind_info->per_backend_data_off);
+
+ LWLockRelease(&entry->lock);
+ dshash_release_lock(hash, entry);
+
+ return stats_data;
+}
+
+/*
+ * Accumulate this process's per-backend stats into the global stats, then
+ * remove the entry from the dshash.
+ * Acquire the kind lock before the dshash partition lock. Snapshots and resets
+ * hold the kind lock while accessing both the global stats and live entries,
+ * so an entry cannot move between them during either operation.
+ *
+ * NB: The entry may belong to an earlier process that used the same ProcNumber.
+ * It must still be accumulated before removal.
+ */
+void
+pgstat_acc_my_per_backend(PgStat_Kind kind, LWLock *lock)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table *hash;
+ void *dst;
+ PgStatShared_PerBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return;
+
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ dst = shared_struct + kind_info->shared_data_off;
+
+ LWLockAcquire(lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &MyProcNumber, true);
+
+ if (entry == NULL)
+ {
+ state->my_entry = NULL;
+ LWLockRelease(lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ if (state->my_entry == entry)
+ state->my_entry = NULL;
+
+ /* Kind-specific accumulation */
+ kind_info->per_backend_acc_cb(dst, entry);
+
+ LWLockRelease(&entry->lock);
+
+ /* Remove the entry */
+ dshash_delete_entry(hash, entry);
+
+ LWLockRelease(lock);
+}
+
+/*
+ * Accumulate all per-backend entries into global stats and delete them.
+ * Called at clean server shutdown before writing the stats file. Acquire the
+ * kind's global lock before starting the dshash scan.
+ */
+void
+pgstat_acc_all_per_backend(PgStat_Kind kind, LWLock *lock)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table *hash;
+ void *dst;
+ dshash_seq_status hstat;
+ PgStatShared_PerBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return;
+
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ dst = shared_struct + kind_info->shared_data_off;
+
+ LWLockAcquire(lock, LW_EXCLUSIVE);
+
+ dshash_seq_init(&hstat, hash, true);
+
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ if (state->my_entry == entry)
+ state->my_entry = NULL;
+
+ /* Kind-specific accumulation */
+ kind_info->per_backend_acc_cb(dst, entry);
+
+ LWLockRelease(&entry->lock);
+ dshash_delete_current(&hstat);
+ }
+
+ dshash_seq_term(&hstat);
+
+ LWLockRelease(lock);
+}
+
static void
pgstat_build_snapshot(void)
{
@@ -1540,6 +2041,11 @@ pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
errdetail("Custom cumulative statistics must be registered while initializing modules in \"%s\".",
"shared_preload_libraries")));
+ if (kind_info->per_backend_data_len != 0)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Per-backend statistics are not supported for custom cumulative statistics.")));
+
/*
* Check some data for fixed-numbered stats.
*/
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 8511c25fd29..3b003f00245 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -168,6 +168,47 @@ StatsShmemRequest(void *arg)
);
}
+/*
+ * Create a dshash for each built-in kind that stores per-backend statistics.
+ * Derive the entry size and handle location from the kind metadata, just as
+ * attachment does.
+ */
+static void
+pgstat_create_per_backend_hashes(dsa_area *dsa, PgStat_ShmemControl *ctl)
+{
+ dshash_parameters params = {
+ sizeof(ProcNumber),
+ 0,
+ dshash_memcmp,
+ dshash_memhash,
+ dshash_memcpy,
+ LWTRANCHE_PGSTATS_HASH
+ };
+
+ for (PgStat_Kind kind = PGSTAT_KIND_BUILTIN_MIN;
+ kind <= PGSTAT_KIND_BUILTIN_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ char *shared_struct;
+ dshash_table_handle *handle_ptr;
+ dshash_table *dsh;
+
+ if (kind_info == NULL || kind_info->per_backend_data_len == 0)
+ continue;
+
+ shared_struct = (char *) ctl + kind_info->shared_ctl_off;
+ handle_ptr = (dshash_table_handle *)
+ (shared_struct + kind_info->per_backend_hash_handle_off);
+
+ params.entry_size = kind_info->per_backend_data_off +
+ kind_info->per_backend_data_len;
+
+ dsh = dshash_create(dsa, ¶ms, NULL);
+ *handle_ptr = dshash_get_hash_table_handle(dsh);
+ dshash_detach(dsh);
+ }
+}
+
/*
* Initialize cumulative statistics system during startup
*/
@@ -210,6 +251,11 @@ StatsShmemInit(void *arg)
/* lift limit set above */
dsa_set_size_limit(dsa, -1);
+ /*
+ * Create per-backend hashes while the local DSA reference is available.
+ */
+ pgstat_create_per_backend_hashes(dsa, ctl);
+
/*
* Postmaster will never access these again, thus free the local
* dsa/dshash references.
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index b0a17691966..340244252c9 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -380,6 +380,26 @@ typedef struct PgStat_KindInfo
*/
void (*snapshot_cb) (void);
+ /*
+ * Per-backend dshash support for built-in fixed-numbered statistics kinds
+ * that also maintain per-backend entries in a dedicated dshash. If
+ * per_backend_data_len is non-zero, the generic infrastructure handles
+ * attach, fetch, accumulate, and snapshot pre-caching automatically.
+ *
+ * Each entry starts with PgStatShared_PerBackendEntry, followed by the
+ * kind-specific statistics payload.
+ */
+ uint32 per_backend_data_off; /* offset of stats data in entry */
+ uint32 per_backend_data_len; /* size of stats data in entry */
+ /* offset of dshash_table_handle in shared struct */
+ uint32 per_backend_hash_handle_off;
+
+ /*
+ * Callback to accumulate one per-backend entry into a destination of the
+ * kind's statistics type. Called with the entry's content lock held.
+ */
+ void (*per_backend_acc_cb) (void *dst, void *entry);
+
/* name of the kind of stats */
const char *const name;
} PgStat_KindInfo;
@@ -479,6 +499,17 @@ typedef struct PgStatShared_SLRU
PgStat_SLRUStats stats[SLRU_NUM_ELEMENTS];
} PgStatShared_SLRU;
+/*
+ * Common header for entries in per-backend statistics dshashes. The
+ * ProcNumber key must be the first field for dshash.
+ */
+typedef struct PgStatShared_PerBackendEntry
+{
+ ProcNumber key;
+ BackendType backend_type;
+ LWLock lock;
+} PgStatShared_PerBackendEntry;
+
typedef struct PgStatShared_Wal
{
/* lock protects ->stats */
@@ -617,6 +648,9 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ /* Per-backend snapshot hash */
+ struct pgstat_per_backend_snapshot_hash *per_backend_stats;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -691,6 +725,15 @@ extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid,
bool *may_free);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+/* Generic per-backend helpers */
+extern dshash_table *pgstat_per_backend_attach(PgStat_Kind kind);
+extern void *pgstat_lock_my_per_backend_entry(PgStat_Kind kind, bool nowait);
+extern void pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash,
+ void *snap);
+extern void *pgstat_fetch_per_backend(PgStat_Kind kind, ProcNumber procnum);
+extern void pgstat_acc_my_per_backend(PgStat_Kind kind, LWLock *lock);
+extern void pgstat_acc_all_per_backend(PgStat_Kind kind, LWLock *lock);
+
/*
* Functions in pgstat_archiver.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 85d989f395d..0b7dbc42e80 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2327,6 +2327,7 @@ PgStatShared_Function
PgStatShared_HashEntry
PgStatShared_IO
PgStatShared_Lock
+PgStatShared_PerBackendEntry
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2351,6 +2352,9 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_Lock
PgStat_LockEntry
+PgStat_PerBackendLocalState
+PgStat_PerBackendSnapshotEntry
+PgStat_PerBackendSnapshotKey
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
--
2.34.1
[text/x-diff] v1-0003-pgstat-move-WAL-statistics-to-new-per-backend-inf.patch (20.7K, ../../anCjoFkGah%2FuB7lw@bdtpg/4-v1-0003-pgstat-move-WAL-statistics-to-new-per-backend-inf.patch)
download | inline diff:
From 753426d6e851c9942985e0bd90583c6de3deaaa8 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 29 Jul 2026 14:13:28 +0000
Subject: [PATCH v1 3/5] pgstat: move WAL statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's WAL, Lock, and IO statistics
together in one variable-numbered entry keyed by ProcNumber. Move WAL
statistics into a dedicated ProcNumber keyed dshash associated with the
fixed WAL statistics kind. The global WAL stats hold data for backends that
have exited, while the dshash holds statistics for live backends.
Build pg_stat_wal snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove WAL counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_wal(). This also makes WAL statistics available for auxiliary
and shared memory-only workers. Update the documentation to describe the new
behavior.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by:
Discussion:
---
doc/src/sgml/monitoring.sgml | 7 +-
src/backend/utils/activity/pgstat.c | 18 ++
src/backend/utils/activity/pgstat_backend.c | 75 --------
src/backend/utils/activity/pgstat_wal.c | 179 +++++++++++++++++---
src/backend/utils/adt/pgstatfuncs.c | 37 +++-
src/include/pgstat.h | 3 +-
src/include/utils/pgstat_internal.h | 22 ++-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 232 insertions(+), 110 deletions(-)
3.6% doc/src/sgml/
78.0% src/backend/utils/activity/
10.2% src/backend/utils/adt/
6.0% src/include/utils/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 099e9b6f4e9..bd5881beb61 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5853,10 +5853,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns WAL statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_wal</structname> view.
- </para>
- <para>
- The function does not return WAL statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher.
</para></entry>
</row>
@@ -5996,7 +5992,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>wal</literal>: Reset all the counters shown in the
- <structname>pg_stat_wal</structname> view.
+ <structname>pg_stat_wal</structname> view, as well as per-backend
+ WAL statistics returned by <function>pg_stat_get_backend_wal</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index efc7d427f22..19a98cdd8b7 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -535,6 +535,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_wal_init_shmem_cb,
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_WalBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_WalStats),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_Wal, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_wal_per_backend_acc_cb,
},
};
@@ -646,6 +651,9 @@ pgstat_before_server_shutdown(int code, Datum arg)
*/
if (code == 0)
{
+ /* Transfer all live per-backend stats before writing the stats file. */
+ pgstat_wal_acc_all_backends();
+
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
}
@@ -689,6 +697,9 @@ pgstat_shutdown_hook(int code, Datum arg)
if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
pgstat_request_entry_refs_gc();
+ /* Accumulate per-backend WAL stats into the global stats */
+ pgstat_wal_acc_backend_cb();
+
pgstat_detach_shmem();
#ifdef USE_ASSERT_CHECKING
@@ -709,6 +720,13 @@ pgstat_initialize(void)
pgstat_attach_shmem();
+ /*
+ * NB: need to accept that there might be stats from an older backend that
+ * used the same proc number. Accumulate them into the global stats before
+ * we start using the entry.
+ */
+ pgstat_wal_acc_backend_cb();
+
/*
* Create and cache per-backend statistics entries here. This also covers
* processes that never call InitPostgres(), such as shared-memory-only
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index b736b2ccc6f..76b970a8c41 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -41,14 +41,6 @@ static PgStat_BackendPending PendingBackendStats;
static bool backend_has_iostats = false;
static bool backend_has_lockstats = false;
-/*
- * WAL usage counters saved from pgWalUsage at the previous call to
- * pgstat_flush_backend(). This is used to calculate how much WAL usage
- * happens between pgstat_flush_backend() calls, by subtracting the
- * previous counters from the current ones.
- */
-static WalUsage prevBackendWalUsage;
-
/*
* Utility routines to report I/O stats for backends, kept here to avoid
* exposing PendingBackendStats to the outside world.
@@ -244,58 +236,6 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
backend_has_iostats = false;
}
-/*
- * To determine whether WAL usage happened.
- */
-static inline bool
-pgstat_backend_wal_have_pending(void)
-{
- return (pgWalUsage.wal_records != prevBackendWalUsage.wal_records);
-}
-
-/*
- * Flush out locally pending backend WAL statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_WalCounters *bktype_shstats;
- WalUsage wal_usage_diff = {0};
-
- /*
- * This function can be called even if nothing at all has happened for WAL
- * statistics. In this case, avoid unnecessarily modifying the stats
- * entry.
- */
- if (!pgstat_backend_wal_have_pending())
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.wal_counters;
-
- /*
- * Calculate how much WAL usage counters were increased by subtracting the
- * previous counters from the current ones.
- */
- WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevBackendWalUsage);
-
-#define WALSTAT_ACC(fld, var_to_add) \
- (bktype_shstats->fld += var_to_add.fld)
- WALSTAT_ACC(wal_buffers_full, wal_usage_diff);
- WALSTAT_ACC(wal_records, wal_usage_diff);
- WALSTAT_ACC(wal_fpi, wal_usage_diff);
- WALSTAT_ACC(wal_bytes, wal_usage_diff);
- WALSTAT_ACC(wal_fpi_bytes, wal_usage_diff);
-#undef WALSTAT_ACC
-
- /*
- * Save the current counters for the subsequent calculation of WAL usage.
- */
- prevBackendWalUsage = pgWalUsage;
-}
-
/*
* Flush out locally pending backend lock statistics. Locking is managed
* by the caller.
@@ -345,11 +285,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
has_pending_data = true;
- /* Some WAL data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_WAL) &&
- pgstat_backend_wal_have_pending())
- has_pending_data = true;
-
/* Some lock data pending? */
if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats)
has_pending_data = true;
@@ -366,9 +301,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_IO)
pgstat_flush_backend_entry_io(entry_ref);
- if (flags & PGSTAT_BACKEND_FLUSH_WAL)
- pgstat_flush_backend_entry_wal(entry_ref);
-
if (flags & PGSTAT_BACKEND_FLUSH_LOCK)
pgstat_flush_backend_entry_lock(entry_ref);
@@ -411,13 +343,6 @@ pgstat_create_backend(ProcNumber procnum)
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
backend_has_iostats = false;
backend_has_lockstats = false;
-
- /*
- * Initialize prevBackendWalUsage with pgWalUsage so that
- * pgstat_backend_flush_cb() can calculate how much pgWalUsage counters
- * are increased by subtracting prevBackendWalUsage from pgWalUsage.
- */
- prevBackendWalUsage = pgWalUsage;
}
/*
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..ece4d91ed70 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -8,6 +8,11 @@
* storage implementation and the details about individual types of
* statistics.
*
+ * WAL statistics use a per-backend dshash to avoid double-counting. Each
+ * backend flushes WAL usage to its own entry in the dshash (keyed by
+ * ProcNumber). The global pg_stat_wal view aggregates the global stats
+ * (which holds stats from exited backends) plus all live per-backend entries.
+ *
* Copyright (c) 2001-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -52,7 +57,6 @@ pgstat_report_wal(bool force)
/* flush wal stats */
(void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
/* flush IO stats */
pgstat_flush_io(nowait);
@@ -84,13 +88,15 @@ pgstat_wal_have_pending(void)
* Calculate how much WAL usage counters have increased by subtracting the
* previous counters from the current ones.
*
+ * Flush WAL usage counters to the per-backend dshash entry.
+ *
* If nowait is true, this function returns true if the lock could not be
* acquired. Otherwise return false.
*/
bool
pgstat_wal_flush_cb(bool nowait)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_WalBackendEntry *entry;
WalUsage wal_usage_diff = {0};
Assert(IsUnderPostmaster || !IsPostmasterEnvironment);
@@ -105,19 +111,18 @@ pgstat_wal_flush_cb(bool nowait)
return false;
/*
- * We don't update the WAL usage portion of the local WalStats elsewhere.
* Calculate how much WAL usage counters were increased by subtracting the
* previous counters from the current ones.
*/
WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevWalUsage);
- if (!nowait)
- LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
- return true;
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_WAL, nowait);
+
+ if (entry == NULL)
+ return nowait;
#define WALSTAT_ACC(fld, var_to_add) \
- (stats_shmem->stats.wal_counters.fld += var_to_add.fld)
+ (entry->stats.wal_counters.fld += var_to_add.fld)
WALSTAT_ACC(wal_records, wal_usage_diff);
WALSTAT_ACC(wal_fpi, wal_usage_diff);
WALSTAT_ACC(wal_bytes, wal_usage_diff);
@@ -125,7 +130,7 @@ pgstat_wal_flush_cb(bool nowait)
WALSTAT_ACC(wal_buffers_full, wal_usage_diff);
#undef WALSTAT_ACC
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&entry->header.lock);
/*
* Save the current counters for the subsequent calculation of WAL usage.
@@ -157,21 +162,157 @@ pgstat_wal_init_shmem_cb(void *stats)
void
pgstat_wal_reset_all_cb(TimestampTz ts)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ dshash_seq_status hstat;
+ PgStatShared_WalBackendEntry *entry;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
+
+ /* Reset all per-backend entries */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.wal_counters, 0, sizeof(PgStat_WalCounters));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
+ }
- LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
- memset(&stats_shmem->stats, 0, sizeof(stats_shmem->stats));
- stats_shmem->stats.stat_reset_timestamp = ts;
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build WAL stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_wal_snapshot_cb(void)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ PgStat_WalStats *snap = &pgStatLocal.snapshot.wal;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_WalStats));
+
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_WAL, hash, snap);
- LWLockAcquire(&stats_shmem->lock, LW_SHARED);
- memcpy(&pgStatLocal.snapshot.wal, &stats_shmem->stats,
- sizeof(pgStatLocal.snapshot.wal));
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&shmem->lock);
+}
+
+/* Macro to accumulate WAL counters from src into dst */
+#define WAL_ACCUMULATE_COUNTERS(dst, src) \
+do { \
+ (dst).wal_records += (src).wal_records; \
+ (dst).wal_fpi += (src).wal_fpi; \
+ (dst).wal_bytes += (src).wal_bytes; \
+ (dst).wal_fpi_bytes += (src).wal_fpi_bytes; \
+ (dst).wal_buffers_full += (src).wal_buffers_full; \
+} while (0)
+
+/*
+ * Accumulate one per-backend WAL entry into a snapshot or the global stats.
+ */
+void
+pgstat_wal_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_WalStats *stats = dst;
+ PgStatShared_WalBackendEntry *e = (PgStatShared_WalBackendEntry *) entry;
+
+ WAL_ACCUMULATE_COUNTERS(stats->wal_counters, e->stats.wal_counters);
+}
+
+/*
+ * Accumulate a backend's WAL stats into the global stats, then
+ * remove the entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_wal_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_WAL, &pgStatLocal.shmem->wal.lock);
+}
+
+/*
+ * Returns per-backend WAL statistics for the given ProcNumber.
+ */
+PgStat_WalStats *
+pgstat_fetch_stat_backend_wal(ProcNumber procnum)
+{
+ return (PgStat_WalStats *) pgstat_fetch_per_backend(PGSTAT_KIND_WAL, procnum);
+}
+
+/*
+ * Reset a backend's WAL stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_wal_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ dshash_table *hash;
+ PgStatShared_WalBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ WAL_ACCUMULATE_COUNTERS(shmem->stats.wal_counters, entry->stats.wal_counters);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats.wal_counters, 0, sizeof(PgStat_WalCounters));
+ entry->stats.stat_reset_timestamp = ts;
+
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
+}
+
+/*
+ * Accumulate all per-backend WAL stats entries into the global stats and remove
+ * them. Called at clean server shutdown to ensure all flushed data is preserved
+ * in the stats file.
+ */
+void
+pgstat_wal_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_WAL, &pgStatLocal.shmem->wal.lock);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 565d0e70768..5b0c819492a 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1707,19 +1707,33 @@ Datum
pg_stat_get_backend_wal(PG_FUNCTION_ARGS)
{
int pid;
- PgStat_Backend *backend_stats;
- PgStat_WalCounters bktype_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ PgStat_WalStats *wal_stats;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, NULL);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
+ PG_RETURN_NULL();
+
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
PG_RETURN_NULL();
- bktype_stats = backend_stats->wal_counters;
+ wal_stats = pgstat_fetch_stat_backend_wal(procnum);
- /* save tuples with data from this PgStat_WalCounters */
- return (pg_stat_wal_build_tuple(bktype_stats, backend_stats->stat_reset_timestamp));
+ if (!wal_stats)
+ PG_RETURN_NULL();
+
+ return (pg_stat_wal_build_tuple(wal_stats->wal_counters,
+ wal_stats->stat_reset_timestamp));
}
/*
@@ -2066,6 +2080,7 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
PGPROC *proc;
PgBackendStatus *beentry;
ProcNumber procNumber;
+ TimestampTz ts;
int backend_pid = PG_GETARG_INT32(0);
proc = BackendPidGetProc(backend_pid);
@@ -2087,6 +2102,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
PG_RETURN_VOID();
+ /*
+ * Accumulate the backend's WAL stats into the global stats, then zero the
+ * entry.
+ */
+ ts = GetCurrentTimestamp();
+ pgstat_wal_reset_backend_cb(procNumber, ts);
+
+ /* Reset IO and Lock stats still in PGSTAT_KIND_BACKEND */
pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
PG_RETURN_VOID();
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 58a44857f13..0ade7f2f053 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -522,7 +522,6 @@ typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
PgStat_BktypeIO io_stats;
- PgStat_WalCounters wal_counters;
PgStat_PendingLock lock_stats;
} PgStat_Backend;
@@ -842,6 +841,8 @@ extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_
extern void pgstat_report_wal(bool force);
extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
+extern PgStat_WalStats *pgstat_fetch_stat_backend_wal(ProcNumber procnum);
+extern void pgstat_wal_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
/*
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 340244252c9..23ed4bdb4cc 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -510,11 +510,25 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for WAL statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_WalBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_WalStats stats;
+} PgStatShared_WalBackendEntry;
+
typedef struct PgStatShared_Wal
{
- /* lock protects ->stats */
LWLock lock;
PgStat_WalStats stats;
+
+ /*
+ * Per-backend dshash, keyed by ProcNumber.
+ */
+ dshash_table_handle backend_hash_handle;
} PgStatShared_Wal;
@@ -749,9 +763,8 @@ extern void pgstat_archiver_snapshot_cb(void);
/* flags for pgstat_flush_backend() */
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL | PGSTAT_BACKEND_FLUSH_LOCK)
+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_LOCK)
extern bool pgstat_flush_backend(bool nowait, uint32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
@@ -890,6 +903,9 @@ extern bool pgstat_wal_flush_cb(bool nowait);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
+extern void pgstat_wal_acc_backend_cb(void);
+extern void pgstat_wal_acc_all_backends(void);
+extern void pgstat_wal_per_backend_acc_cb(void *dst, void *entry);
/*
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 0b7dbc42e80..5a4be198ada 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2333,6 +2333,7 @@ PgStatShared_ReplSlot
PgStatShared_SLRU
PgStatShared_Subscription
PgStatShared_Wal
+PgStatShared_WalBackendEntry
PgStat_ArchiverStats
PgStat_Backend
PgStat_BackendPending
--
2.34.1
[text/x-diff] v1-0004-pgstat-move-Lock-statistics-to-new-per-backend-in.patch (21.2K, ../../anCjoFkGah%2FuB7lw@bdtpg/5-v1-0004-pgstat-move-Lock-statistics-to-new-per-backend-in.patch)
download | inline diff:
From a105e6c596907d2def426ac05d1b0c828d3ac50a Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:25:10 +0000
Subject: [PATCH v1 4/5] pgstat: move Lock statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's Lock and IO statistics together in one
variable-numbered entry keyed by ProcNumber. Move Lock statistics into a dedicated
ProcNumber keyed dshash associated with the fixed Lock statistics kind. The global
Lock stats hold data for backends that have exited, while the dshash holds
statistics for live backends.
Build pg_stat_lock snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove Lock counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_lock(). This also makes Lock statistics available for
auxiliary and shared memory-only workers. Update the documentation to describe the
new behavior.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by:
Discussion:
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/utils/activity/pgstat.c | 10 +-
src/backend/utils/activity/pgstat_backend.c | 72 --------
src/backend/utils/activity/pgstat_lock.c | 183 +++++++++++++++++---
src/backend/utils/adt/pgstatfuncs.c | 34 +++-
src/include/pgstat.h | 12 +-
src/include/utils/pgstat_internal.h | 21 ++-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 218 insertions(+), 123 deletions(-)
3.6% doc/src/sgml/
74.4% src/backend/utils/activity/
9.8% src/backend/utils/adt/
6.9% src/include/utils/
4.7% src/include/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index bd5881beb61..d3618e0b920 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5834,10 +5834,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns lock statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_lock</structname> view.
- </para>
- <para>
- The function does not return lock statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher.
</para></entry>
</row>
@@ -5974,7 +5970,9 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>lock</literal>: Reset all the counters shown in the
- <structname>pg_stat_lock</structname> view.
+ <structname>pg_stat_lock</structname> view, as well as per-backend
+ lock statistics returned by
+ <function>pg_stat_get_backend_lock</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 19a98cdd8b7..2234bdb6857 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -500,6 +500,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_lock_init_shmem_cb,
.reset_all_cb = pgstat_lock_reset_all_cb,
.snapshot_cb = pgstat_lock_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_LockBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_Lock),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_Lock, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_lock_per_backend_acc_cb,
},
[PGSTAT_KIND_SLRU] = {
@@ -653,6 +658,7 @@ pgstat_before_server_shutdown(int code, Datum arg)
{
/* Transfer all live per-backend stats before writing the stats file. */
pgstat_wal_acc_all_backends();
+ pgstat_lock_acc_all_backends();
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
@@ -697,8 +703,9 @@ pgstat_shutdown_hook(int code, Datum arg)
if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
pgstat_request_entry_refs_gc();
- /* Accumulate per-backend WAL stats into the global stats */
+ /* Accumulate per-backend stats into the global stats */
pgstat_wal_acc_backend_cb();
+ pgstat_lock_acc_backend_cb();
pgstat_detach_shmem();
@@ -726,6 +733,7 @@ pgstat_initialize(void)
* we start using the entry.
*/
pgstat_wal_acc_backend_cb();
+ pgstat_lock_acc_backend_cb();
/*
* Create and cache per-backend statistics entries here. This also covers
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 76b970a8c41..8dda2cd88c1 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -39,7 +39,6 @@
*/
static PgStat_BackendPending PendingBackendStats;
static bool backend_has_iostats = false;
-static bool backend_has_lockstats = false;
/*
* Utility routines to report I/O stats for backends, kept here to avoid
@@ -79,39 +78,6 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
pgstat_report_fixed = true;
}
-/*
- * Utility routines to report lock stats for backends, kept here to avoid
- * exposing PendingBackendStats to the outside world.
- */
-void
-pgstat_count_backend_lock_waits(uint8 locktag_type, PgStat_Counter usecs)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(locktag_type <= LOCKTAG_LAST_TYPE);
-
- PendingBackendStats.pending_lock.stats[locktag_type].waits++;
- PendingBackendStats.pending_lock.stats[locktag_type].wait_time += usecs;
-
- backend_has_lockstats = true;
- pgstat_report_fixed = true;
-}
-
-void
-pgstat_count_backend_lock_fastpath_exceeded(uint8 locktag_type)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(locktag_type <= LOCKTAG_LAST_TYPE);
-
- PendingBackendStats.pending_lock.stats[locktag_type].fastpath_exceeded++;
-
- backend_has_lockstats = true;
- pgstat_report_fixed = true;
-}
-
/*
* Returns statistics of a backend by proc number.
*/
@@ -236,36 +202,6 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
backend_has_iostats = false;
}
-/*
- * Flush out locally pending backend lock statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_lock(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_PendingLock *bktype_shstats;
-
- if (!backend_has_lockstats)
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.lock_stats;
-
- for (int i = 0; i <= LOCKTAG_LAST_TYPE; i++)
- {
-#define LOCKSTAT_ACC(fld) \
- (bktype_shstats->stats[i].fld += PendingBackendStats.pending_lock.stats[i].fld)
- LOCKSTAT_ACC(waits);
- LOCKSTAT_ACC(wait_time);
- LOCKSTAT_ACC(fastpath_exceeded);
-#undef LOCKSTAT_ACC
- }
-
- MemSet(&PendingBackendStats.pending_lock, 0, sizeof(PgStat_PendingLock));
- backend_has_lockstats = false;
-}
-
/*
* Flush out locally pending backend statistics
*
@@ -285,10 +221,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
has_pending_data = true;
- /* Some lock data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats)
- has_pending_data = true;
-
if (!has_pending_data)
return false;
@@ -301,9 +233,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_IO)
pgstat_flush_backend_entry_io(entry_ref);
- if (flags & PGSTAT_BACKEND_FLUSH_LOCK)
- pgstat_flush_backend_entry_lock(entry_ref);
-
pgstat_unlock_entry(entry_ref);
return false;
@@ -342,7 +271,6 @@ pgstat_create_backend(ProcNumber procnum)
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
backend_has_iostats = false;
- backend_has_lockstats = false;
}
/*
diff --git a/src/backend/utils/activity/pgstat_lock.c b/src/backend/utils/activity/pgstat_lock.c
index c20c7599683..5aa380a6243 100644
--- a/src/backend/utils/activity/pgstat_lock.c
+++ b/src/backend/utils/activity/pgstat_lock.c
@@ -8,6 +8,12 @@
* access / storage implementation and the details about individual types
* of statistics.
*
+ * Lock statistics use a per-backend dshash to avoid double-counting. Each
+ * backend flushes lock stats (waits, wait_time, fastpath_exceeded per lock
+ * tag type) to its own entry in the dshash. The global pg_stat_lock view
+ * aggregates the global stats (which holds stats from exited backends) plus
+ * all live per-backend entries.
+ *
* Copyright (c) 2021-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -40,7 +46,7 @@ pgstat_lock_flush(bool nowait)
}
/*
- * Flush out locally pending lock statistics
+ * Flush out locally pending lock statistics to the per-backend dshash entry.
*
* If no stats have been recorded, this function returns false.
*
@@ -50,31 +56,27 @@ pgstat_lock_flush(bool nowait)
bool
pgstat_lock_flush_cb(bool nowait)
{
- LWLock *lckstat_lock;
- PgStatShared_Lock *shstats;
+ PgStatShared_LockBackendEntry *entry;
if (!have_lockstats)
return false;
- shstats = &pgStatLocal.shmem->lock;
- lckstat_lock = &shstats->lock;
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_LOCK, nowait);
- if (!nowait)
- LWLockAcquire(lckstat_lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(lckstat_lock, LW_EXCLUSIVE))
- return true;
+ if (entry == NULL)
+ return nowait;
for (int i = 0; i <= LOCKTAG_LAST_TYPE; i++)
{
#define LOCKSTAT_ACC(fld) \
- (shstats->stats.stats[i].fld += PendingLockStats.stats[i].fld)
+ (entry->stats.stats[i].fld += PendingLockStats.stats[i].fld)
LOCKSTAT_ACC(waits);
LOCKSTAT_ACC(wait_time);
LOCKSTAT_ACC(fastpath_exceeded);
#undef LOCKSTAT_ACC
}
- LWLockRelease(lckstat_lock);
+ LWLockRelease(&entry->header.lock);
memset(&PendingLockStats, 0, sizeof(PendingLockStats));
have_lockstats = false;
@@ -93,28 +95,125 @@ pgstat_lock_init_shmem_cb(void *stats)
void
pgstat_lock_reset_all_cb(TimestampTz ts)
{
- LWLock *lckstat_lock = &pgStatLocal.shmem->lock.lock;
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ dshash_seq_status hstat;
+ PgStatShared_LockBackendEntry *entry;
+ dshash_table *hash;
- LWLockAcquire(lckstat_lock, LW_EXCLUSIVE);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
- pgStatLocal.shmem->lock.stats.stat_reset_timestamp = ts;
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
- memset(pgStatLocal.shmem->lock.stats.stats, 0,
- sizeof(pgStatLocal.shmem->lock.stats.stats));
+ /* Reset all per-backend entries, since they contribute to the global view */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.stats, 0, sizeof(entry->stats.stats));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
+ }
- LWLockRelease(lckstat_lock);
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build lock stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_lock_snapshot_cb(void)
{
- LWLock *lckstat_lock = &pgStatLocal.shmem->lock.lock;
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ PgStat_Lock *snap = &pgStatLocal.snapshot.lock;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
+
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_Lock));
+
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_LOCK, hash, snap);
+
+ LWLockRelease(&shmem->lock);
+}
+
+/*
+ * Accumulate one per-backend lock entry into a snapshot or the global stats.
+ */
+void
+pgstat_lock_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_Lock *stats = dst;
+ PgStatShared_LockBackendEntry *e = (PgStatShared_LockBackendEntry *) entry;
+
+ for (int j = 0; j <= LOCKTAG_LAST_TYPE; j++)
+ {
+ stats->stats[j].waits += e->stats.stats[j].waits;
+ stats->stats[j].wait_time += e->stats.stats[j].wait_time;
+ stats->stats[j].fastpath_exceeded += e->stats.stats[j].fastpath_exceeded;
+ }
+}
+
+/* Macro to accumulate lock counters from src into dst */
+#define LOCK_ACCUMULATE_COUNTERS(dst, src) \
+do { \
+ for (int _i = 0; _i <= LOCKTAG_LAST_TYPE; _i++) \
+ { \
+ (dst)[_i].waits += (src)[_i].waits; \
+ (dst)[_i].wait_time += (src)[_i].wait_time; \
+ (dst)[_i].fastpath_exceeded += (src)[_i].fastpath_exceeded; \
+ } \
+} while (0)
- LWLockAcquire(lckstat_lock, LW_SHARED);
+/*
+ * Accumulate a backend's lock stats into the global stats, then
+ * remove the entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_lock_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_LOCK, &pgStatLocal.shmem->lock.lock);
+}
- pgStatLocal.snapshot.lock = pgStatLocal.shmem->lock.stats;
+/*
+ * Accumulate all per-backend lock stats entries into the global stats and remove
+ * them. Called at clean server shutdown to ensure all flushed data is preserved
+ * in the stats file.
+ */
+void
+pgstat_lock_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_LOCK, &pgStatLocal.shmem->lock.lock);
+}
- LWLockRelease(lckstat_lock);
+/*
+ * Returns per-backend lock statistics for the given ProcNumber.
+ */
+PgStat_Lock *
+pgstat_fetch_stat_backend_lock(ProcNumber procnum)
+{
+ return (PgStat_Lock *) pgstat_fetch_per_backend(PGSTAT_KIND_LOCK, procnum);
}
/*
@@ -131,8 +230,6 @@ pgstat_count_lock_fastpath_exceeded(uint8 locktag_type)
PendingLockStats.stats[locktag_type].fastpath_exceeded++;
have_lockstats = true;
pgstat_report_fixed = true;
-
- pgstat_count_backend_lock_fastpath_exceeded(locktag_type);
}
/*
@@ -149,6 +246,44 @@ pgstat_count_lock_waits(uint8 locktag_type, PgStat_Counter usecs)
PendingLockStats.stats[locktag_type].wait_time += usecs;
have_lockstats = true;
pgstat_report_fixed = true;
+}
+
+/*
+ * Reset a backend's lock stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_lock_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ dshash_table *hash;
+ PgStatShared_LockBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ LOCK_ACCUMULATE_COUNTERS(shmem->stats.stats, entry->stats.stats);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats.stats, 0, sizeof(entry->stats.stats));
+ entry->stats.stat_reset_timestamp = ts;
- pgstat_count_backend_lock_waits(locktag_type, usecs);
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 5b0c819492a..c26b50e0d28 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1810,19 +1810,36 @@ pg_stat_get_backend_lock(PG_FUNCTION_ARGS)
{
int pid;
ReturnSetInfo *rsinfo;
- PgStat_Backend *backend_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ PgStat_Lock *lock_stats;
InitMaterializedSRF(fcinfo, 0);
rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, NULL);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
return (Datum) 0;
- pg_stat_lock_build_tuples(rsinfo, backend_stats->lock_stats.stats,
- backend_stats->stat_reset_timestamp);
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
+ return (Datum) 0;
+
+ lock_stats = pgstat_fetch_stat_backend_lock(procnum);
+
+ if (!lock_stats)
+ return (Datum) 0;
+
+ pg_stat_lock_build_tuples(rsinfo, lock_stats->stats,
+ lock_stats->stat_reset_timestamp);
return (Datum) 0;
}
@@ -2103,13 +2120,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
/*
- * Accumulate the backend's WAL stats into the global stats, then zero the
- * entry.
+ * Accumulate the backend's WAL and lock stats into the global stats, then
+ * zero the entries.
*/
ts = GetCurrentTimestamp();
pgstat_wal_reset_backend_cb(procNumber, ts);
+ pgstat_lock_reset_backend_cb(procNumber, ts);
- /* Reset IO and Lock stats still in PGSTAT_KIND_BACKEND */
+ /* Reset IO stats still in PGSTAT_KIND_BACKEND */
pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
PG_RETURN_VOID();
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 0ade7f2f053..3911ee56943 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -522,7 +522,6 @@ typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
PgStat_BktypeIO io_stats;
- PgStat_PendingLock lock_stats;
} PgStat_Backend;
/* ---------
@@ -535,12 +534,6 @@ typedef struct PgStat_BackendPending
* Backend statistics store the same amount of IO data as PGSTAT_KIND_IO.
*/
PgStat_PendingIO pending_io;
-
- /*
- * Backend statistics store the same amount of lock data as
- * PGSTAT_KIND_LOCK.
- */
- PgStat_PendingLock pending_lock;
} PgStat_BackendPending;
/*
@@ -593,9 +586,6 @@ extern void pgstat_count_backend_io_op(IOObject io_object,
IOOp io_op, uint32 cnt,
uint64 bytes);
-/* used by pgstat_lock.c for lock stats tracked in backends */
-extern void pgstat_count_backend_lock_waits(uint8 locktag_type, PgStat_Counter usecs);
-extern void pgstat_count_backend_lock_fastpath_exceeded(uint8 locktag_type);
extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber);
extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid,
@@ -652,6 +642,8 @@ extern void pgstat_count_lock_fastpath_exceeded(uint8 locktag_type);
extern void pgstat_count_lock_waits(uint8 locktag_type,
PgStat_Counter usecs);
extern PgStat_Lock *pgstat_fetch_stat_lock(void);
+extern PgStat_Lock *pgstat_fetch_stat_backend_lock(ProcNumber procnum);
+extern void pgstat_lock_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
/*
* Functions in pgstat_database.c
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 23ed4bdb4cc..9a765eca359 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -487,9 +487,12 @@ typedef struct PgStatShared_IO
typedef struct PgStatShared_Lock
{
- /* lock protects ->stats */
+ /* lock protects ->stats (global stats) */
LWLock lock;
PgStat_Lock stats;
+
+ /* Per-backend dshash, keyed by ProcNumber */
+ dshash_table_handle backend_hash_handle;
} PgStatShared_Lock;
typedef struct PgStatShared_SLRU
@@ -510,6 +513,16 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for lock statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_LockBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_Lock stats;
+} PgStatShared_LockBackendEntry;
+
/*
* Per-backend entry for WAL statistics, stored in a dshash keyed by
* ProcNumber.
@@ -763,8 +776,7 @@ extern void pgstat_archiver_snapshot_cb(void);
/* flags for pgstat_flush_backend() */
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_LOCK)
+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO)
extern bool pgstat_flush_backend(bool nowait, uint32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
@@ -830,6 +842,9 @@ extern bool pgstat_lock_flush_cb(bool nowait);
extern void pgstat_lock_init_shmem_cb(void *stats);
extern void pgstat_lock_reset_all_cb(TimestampTz ts);
extern void pgstat_lock_snapshot_cb(void);
+extern void pgstat_lock_acc_backend_cb(void);
+extern void pgstat_lock_acc_all_backends(void);
+extern void pgstat_lock_per_backend_acc_cb(void *dst, void *entry);
/*
* Functions in pgstat_relation.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 5a4be198ada..d89fccf482d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2327,6 +2327,7 @@ PgStatShared_Function
PgStatShared_HashEntry
PgStatShared_IO
PgStatShared_Lock
+PgStatShared_LockBackendEntry
PgStatShared_PerBackendEntry
PgStatShared_Relation
PgStatShared_ReplSlot
--
2.34.1
[text/x-diff] v1-0005-pgstat-move-IO-statistics-to-new-per-backend-infr.patch (42.9K, ../../anCjoFkGah%2FuB7lw@bdtpg/6-v1-0005-pgstat-move-IO-statistics-to-new-per-backend-infr.patch)
download | inline diff:
From c5e8cffb9c42c0c8ebfe1c49ad58a099b97a4566 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:26:07 +0000
Subject: [PATCH v1 5/5] pgstat: move IO statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's IO statistics in one variable-numbered
entry keyed by ProcNumber. Move IO statistics into a dedicated ProcNumber keyed
dshash associated with the fixed IO statistics kind. The global IO stats hold data
for backends that have exited, while the dshash holds statistics for live backends.
Build pg_stat_io snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove IO counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_io(). This also makes IO statistics available for auxiliary
and shared memory-only workers. Update the documentation to describe the
resulting new behavior.
After WAL, Lock, and IO have moved, PGSTAT_KIND_BACKEND contains no data.
Remove that kind and pgstat_backend.c together with their build entries, shared
and pending structures, callbacks, flags, and call sites.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by:
Discussion:
---
doc/src/sgml/monitoring.sgml | 9 +-
src/backend/replication/walsender.c | 2 -
src/backend/utils/activity/Makefile | 1 -
src/backend/utils/activity/backend_status.c | 4 -
src/backend/utils/activity/meson.build | 1 -
src/backend/utils/activity/pgstat.c | 29 +-
src/backend/utils/activity/pgstat_backend.c | 332 -------------------
src/backend/utils/activity/pgstat_io.c | 225 ++++++++++---
src/backend/utils/activity/pgstat_relation.c | 2 -
src/backend/utils/activity/pgstat_shmem.c | 9 +-
src/backend/utils/activity/pgstat_wal.c | 1 -
src/backend/utils/adt/pgstatfuncs.c | 47 +--
src/include/pgstat.h | 53 +--
src/include/utils/pgstat_internal.h | 41 +--
src/include/utils/pgstat_kind.h | 15 +-
src/test/regress/expected/stats.out | 40 ++-
src/test/regress/sql/stats.sql | 11 +-
src/tools/pgindent/typedefs.list | 5 +-
18 files changed, 282 insertions(+), 545 deletions(-)
70.6% src/backend/utils/activity/
6.0% src/backend/utils/adt/
6.3% src/include/utils/
5.5% src/include/
6.1% src/test/regress/expected/
3.2% src/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index d3618e0b920..e83a68b686e 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5797,12 +5797,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns I/O statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_io</structname> view.
- </para>
- <para>
- The function does not return I/O statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher
- as they are already visible in the <structname>pg_stat_io</structname>
- view and there is only one of each.
</para></entry>
</row>
@@ -5964,7 +5958,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>io</literal>: Reset all the counters shown in the
- <structname>pg_stat_io</structname> view.
+ <structname>pg_stat_io</structname> view, as well as per-backend
+ I/O statistics returned by <function>pg_stat_get_backend_io</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c65dd324325..6d4c15cad4c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -2083,7 +2083,6 @@ WalSndWaitForWal(XLogRecPtr loc)
WALSENDER_STATS_FLUSH_INTERVAL))
{
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
last_flush = now;
}
@@ -3177,7 +3176,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
WALSENDER_STATS_FLUSH_INTERVAL))
{
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
last_flush = now;
}
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index 5fed953c28a..015e6e05978 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -20,7 +20,6 @@ OBJS = \
backend_status.o \
pgstat.o \
pgstat_archiver.o \
- pgstat_backend.o \
pgstat_bgwriter.o \
pgstat_checkpointer.o \
pgstat_database.o \
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index d685fc5cd87..7088fc09651 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -458,10 +458,6 @@ pgstat_bestart_final(void)
PGSTAT_END_WRITE_ACTIVITY(beentry);
- /* Create the backend statistics entry */
- if (pgstat_tracks_backend_bktype(MyBackendType))
- pgstat_create_backend(MyProcNumber);
-
/* Update app name to current GUC setting */
if (application_name)
pgstat_report_appname(application_name);
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 470b5dac402..afbc5a625ea 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -5,7 +5,6 @@ backend_sources += files(
'backend_status.c',
'pgstat.c',
'pgstat_archiver.c',
- 'pgstat_backend.c',
'pgstat_bgwriter.c',
'pgstat_checkpointer.c',
'pgstat_database.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 2234bdb6857..6af5f2ce6ab 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -77,7 +77,6 @@
*
* Each statistics kind is handled in a dedicated file:
* - pgstat_archiver.c
- * - pgstat_backend.c
* - pgstat_bgwriter.c
* - pgstat_checkpointer.c
* - pgstat_database.c
@@ -402,22 +401,6 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_timestamp_cb = pgstat_subscription_reset_timestamp_cb,
},
- [PGSTAT_KIND_BACKEND] = {
- .name = "backend",
-
- .fixed_amount = false,
- .write_to_file = false,
-
- .accessed_across_databases = true,
-
- .shared_size = sizeof(PgStatShared_Backend),
- .shared_data_off = offsetof(PgStatShared_Backend, stats),
- .shared_data_len = sizeof(((PgStatShared_Backend *) 0)->stats),
-
- .flush_static_cb = pgstat_backend_flush_cb,
- .reset_timestamp_cb = pgstat_backend_reset_timestamp_cb,
- },
-
/* stats for fixed-numbered (mostly 1) objects */
[PGSTAT_KIND_ARCHIVER] = {
@@ -483,6 +466,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_io_init_shmem_cb,
.reset_all_cb = pgstat_io_reset_all_cb,
.snapshot_cb = pgstat_io_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_IOBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_BackendIO),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_IO, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_io_per_backend_acc_cb,
},
[PGSTAT_KIND_LOCK] = {
@@ -659,6 +647,7 @@ pgstat_before_server_shutdown(int code, Datum arg)
/* Transfer all live per-backend stats before writing the stats file. */
pgstat_wal_acc_all_backends();
pgstat_lock_acc_all_backends();
+ pgstat_io_acc_all_backends();
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
@@ -699,13 +688,10 @@ pgstat_shutdown_hook(int code, Datum arg)
Assert(dlist_is_empty(&pgStatPending));
dlist_init(&pgStatPending);
- /* drop the backend stats entry */
- if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
- pgstat_request_entry_refs_gc();
-
/* Accumulate per-backend stats into the global stats */
pgstat_wal_acc_backend_cb();
pgstat_lock_acc_backend_cb();
+ pgstat_io_acc_backend_cb();
pgstat_detach_shmem();
@@ -734,6 +720,7 @@ pgstat_initialize(void)
*/
pgstat_wal_acc_backend_cb();
pgstat_lock_acc_backend_cb();
+ pgstat_io_acc_backend_cb();
/*
* Create and cache per-backend statistics entries here. This also covers
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
deleted file mode 100644
index 8dda2cd88c1..00000000000
--- a/src/backend/utils/activity/pgstat_backend.c
+++ /dev/null
@@ -1,332 +0,0 @@
-/* -------------------------------------------------------------------------
- *
- * pgstat_backend.c
- * Implementation of backend statistics.
- *
- * This file contains the implementation of backend statistics. It is kept
- * separate from pgstat.c to enforce the line between the statistics access /
- * storage implementation and the details about individual types of
- * statistics.
- *
- * This statistics kind uses a proc number as object ID for the hash table
- * of pgstats. Entries are created each time a process is spawned, and are
- * dropped when the process exits. These are not written to the pgstats file
- * on disk. Pending statistics are managed without direct interactions with
- * PgStat_EntryRef->pending, relying on PendingBackendStats instead so as it
- * is possible to report data within critical sections.
- *
- * Copyright (c) 2001-2026, PostgreSQL Global Development Group
- *
- * IDENTIFICATION
- * src/backend/utils/activity/pgstat_backend.c
- * -------------------------------------------------------------------------
- */
-
-#include "postgres.h"
-
-#include "access/xlog.h"
-#include "executor/instrument.h"
-#include "storage/bufmgr.h"
-#include "storage/proc.h"
-#include "storage/procarray.h"
-#include "utils/memutils.h"
-#include "utils/pgstat_internal.h"
-
-/*
- * Backend statistics counts waiting to be flushed out. These counters may be
- * reported within critical sections so we use static memory in order to avoid
- * memory allocation.
- */
-static PgStat_BackendPending PendingBackendStats;
-static bool backend_has_iostats = false;
-
-/*
- * Utility routines to report I/O stats for backends, kept here to avoid
- * exposing PendingBackendStats to the outside world.
- */
-void
-pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
- IOOp io_op, instr_time io_time)
-{
- Assert(track_io_timing || track_wal_io_timing);
-
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
-
- INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
- io_time);
-
- backend_has_iostats = true;
- pgstat_report_fixed = true;
-}
-
-void
-pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
- IOOp io_op, uint32 cnt, uint64 bytes)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
-
- PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
- PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
-
- backend_has_iostats = true;
- pgstat_report_fixed = true;
-}
-
-/*
- * Returns statistics of a backend by proc number.
- */
-PgStat_Backend *
-pgstat_fetch_stat_backend(ProcNumber procNumber)
-{
- PgStat_Backend *backend_entry;
-
- backend_entry = (PgStat_Backend *) pgstat_fetch_entry(PGSTAT_KIND_BACKEND,
- InvalidOid, procNumber,
- NULL);
-
- return backend_entry;
-}
-
-/*
- * Returns statistics of a backend by pid.
- *
- * This routine includes sanity checks to ensure that the backend exists and
- * is running. "bktype" can be optionally defined to return the BackendType
- * of the backend whose statistics are returned.
- */
-PgStat_Backend *
-pgstat_fetch_stat_backend_by_pid(int pid, BackendType *bktype)
-{
- PGPROC *proc;
- PgBackendStatus *beentry;
- ProcNumber procNumber;
- PgStat_Backend *backend_stats;
-
- proc = BackendPidGetProc(pid);
- if (bktype)
- *bktype = B_INVALID;
-
- /* this could be an auxiliary process */
- if (!proc)
- proc = AuxiliaryPidGetProc(pid);
-
- if (!proc)
- return NULL;
-
- procNumber = GetNumberFromPGProc(proc);
-
- beentry = pgstat_get_beentry_by_proc_number(procNumber);
- if (!beentry)
- return NULL;
-
- /* check if the backend type tracks statistics */
- if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
- return NULL;
-
- /* if PID does not match, leave */
- if (beentry->st_procpid != pid)
- return NULL;
-
- if (bktype)
- *bktype = beentry->st_backendType;
-
- /*
- * Retrieve the entry. Note that "beentry" may be freed depending on the
- * value of stats_fetch_consistency, so do not access it from this point.
- */
- backend_stats = pgstat_fetch_stat_backend(procNumber);
- if (!backend_stats)
- {
- if (bktype)
- *bktype = B_INVALID;
- return NULL;
- }
-
- return backend_stats;
-}
-
-/*
- * Flush out locally pending backend IO statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_BktypeIO *bktype_shstats;
- PgStat_PendingIO pending_io;
-
- /*
- * This function can be called even if nothing at all has happened for IO
- * statistics. In this case, avoid unnecessarily modifying the stats
- * entry.
- */
- if (!backend_has_iostats)
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.io_stats;
- pending_io = PendingBackendStats.pending_io;
-
- for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
- {
- for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
- {
- for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
- {
- instr_time time;
-
- bktype_shstats->counts[io_object][io_context][io_op] +=
- pending_io.counts[io_object][io_context][io_op];
- bktype_shstats->bytes[io_object][io_context][io_op] +=
- pending_io.bytes[io_object][io_context][io_op];
- time = pending_io.pending_times[io_object][io_context][io_op];
-
- bktype_shstats->times[io_object][io_context][io_op] +=
- INSTR_TIME_GET_MICROSEC(time);
- }
- }
- }
-
- /*
- * Clear out the statistics buffer, so it can be re-used.
- */
- MemSet(&PendingBackendStats.pending_io, 0, sizeof(PgStat_PendingIO));
-
- backend_has_iostats = false;
-}
-
-/*
- * Flush out locally pending backend statistics
- *
- * "flags" parameter controls which statistics to flush. Returns true
- * if some statistics could not be flushed due to lock contention.
- */
-bool
-pgstat_flush_backend(bool nowait, uint32 flags)
-{
- PgStat_EntryRef *entry_ref;
- bool has_pending_data = false;
-
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return false;
-
- /* Some IO data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
- has_pending_data = true;
-
- if (!has_pending_data)
- return false;
-
- entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid,
- MyProcNumber, nowait);
- if (!entry_ref)
- return true;
-
- /* Flush requested statistics */
- if (flags & PGSTAT_BACKEND_FLUSH_IO)
- pgstat_flush_backend_entry_io(entry_ref);
-
- pgstat_unlock_entry(entry_ref);
-
- return false;
-}
-
-/*
- * Callback to flush out locally pending backend statistics.
- *
- * If some stats could not be flushed due to lock contention, return true.
- */
-bool
-pgstat_backend_flush_cb(bool nowait)
-{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
-}
-
-/*
- * Create backend statistics entry for proc number.
- */
-void
-pgstat_create_backend(ProcNumber procnum)
-{
- PgStat_EntryRef *entry_ref;
- PgStatShared_Backend *shstatent;
-
- entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid,
- procnum, false);
- shstatent = (PgStatShared_Backend *) entry_ref->shared_stats;
-
- /*
- * NB: need to accept that there might be stats from an older backend,
- * e.g. if we previously used this proc number.
- */
- memset(&shstatent->stats, 0, sizeof(shstatent->stats));
- pgstat_unlock_entry(entry_ref);
-
- MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
- backend_has_iostats = false;
-}
-
-/*
- * Backend statistics are not collected for all BackendTypes.
- *
- * The following BackendTypes do not participate in the backend stats
- * subsystem:
- * - The same and for the same reasons as in pgstat_tracks_io_bktype().
- * - B_BG_WRITER, B_CHECKPOINTER, B_STARTUP and B_AUTOVAC_LAUNCHER because their
- * I/O stats are already visible in pg_stat_io and there is only one of those.
- *
- * Function returns true if BackendType participates in the backend stats
- * subsystem and false if it does not.
- *
- * When adding a new BackendType, also consider adding relevant restrictions to
- * pgstat_tracks_io_object() and pgstat_tracks_io_op().
- */
-bool
-pgstat_tracks_backend_bktype(BackendType bktype)
-{
- /*
- * List every type so that new backend types trigger a warning about
- * needing to adjust this switch.
- */
- switch (bktype)
- {
- case B_INVALID:
- case B_AUTOVAC_LAUNCHER:
- case B_DEAD_END_BACKEND:
- case B_ARCHIVER:
- case B_LOGGER:
- case B_BG_WRITER:
- case B_CHECKPOINTER:
- case B_IO_WORKER:
- case B_STARTUP:
- case B_DATACHECKSUMSWORKER_LAUNCHER:
- case B_DATACHECKSUMSWORKER_WORKER:
- return false;
-
- case B_AUTOVAC_WORKER:
- case B_BACKEND:
- case B_BG_WORKER:
- case B_STANDALONE_BACKEND:
- case B_SLOTSYNC_WORKER:
- case B_WAL_RECEIVER:
- case B_WAL_SENDER:
- case B_WAL_SUMMARIZER:
- case B_WAL_WRITER:
- return true;
- }
-
- return false;
-}
-
-void
-pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts)
-{
- ((PgStatShared_Backend *) header)->stats.stat_reset_timestamp = ts;
-}
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 8ec1aad5078..5e65bc5cb4b 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -7,6 +7,11 @@
* from pgstat.c to enforce the line between the statistics access / storage
* implementation and the details about individual types of statistics.
*
+ * IO statistics use a per-backend dshash to avoid double-counting. Each
+ * process flushes IO stats to its own entry in the dshash (keyed by
+ * ProcNumber). The global pg_stat_io view aggregates the global stats
+ * (which holds stats from exited processes) plus all live per-backend entries.
+ *
* Copyright (c) 2021-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -68,9 +73,6 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
PendingIOStats.counts[io_object][io_context][io_op] += cnt;
PendingIOStats.bytes[io_object][io_context][io_op] += bytes;
- /* Add the per-backend counts */
- pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
-
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -143,10 +145,6 @@ pgstat_count_io_op_time(IOObject io_object, IOContext io_context, IOOp io_op,
INSTR_TIME_ADD(PendingIOStats.pending_times[io_object][io_context][io_op],
io_time);
-
- /* Add the per-backend count */
- pgstat_count_backend_io_op_time(io_object, io_context, io_op,
- io_time);
}
pgstat_count_io_op(io_object, io_context, io_op, cnt, bytes);
@@ -170,7 +168,7 @@ pgstat_flush_io(bool nowait)
}
/*
- * Flush out locally pending IO statistics
+ * Flush out locally pending IO statistics to the per-backend dshash entry.
*
* If no stats have been recorded, this function returns false.
*
@@ -180,20 +178,18 @@ pgstat_flush_io(bool nowait)
bool
pgstat_io_flush_cb(bool nowait)
{
- LWLock *bktype_lock;
+ PgStatShared_IOBackendEntry *entry;
PgStat_BktypeIO *bktype_shstats;
if (!have_iostats)
return false;
- bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
- bktype_shstats =
- &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_IO, nowait);
- if (!nowait)
- LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
- return true;
+ if (entry == NULL)
+ return nowait;
+
+ bktype_shstats = &entry->stats.stats;
for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
{
@@ -217,12 +213,9 @@ pgstat_io_flush_cb(bool nowait)
}
}
- Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
-
- LWLockRelease(bktype_lock);
+ LWLockRelease(&entry->header.lock);
memset(&PendingIOStats, 0, sizeof(PendingIOStats));
-
have_iostats = false;
return false;
@@ -271,55 +264,70 @@ pgstat_io_init_shmem_cb(void *stats)
{
PgStatShared_IO *stat_shmem = (PgStatShared_IO *) stats;
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- LWLockInitialize(&stat_shmem->locks[i], LWTRANCHE_PGSTATS_DATA);
+ LWLockInitialize(&stat_shmem->lock, LWTRANCHE_PGSTATS_DATA);
}
void
pgstat_io_reset_all_cb(TimestampTz ts)
{
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- {
- LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
- PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ dshash_seq_status hstat;
+ PgStatShared_IOBackendEntry *entry;
+ dshash_table *hash;
- LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
- /*
- * Use the lock in the first BackendType's PgStat_BktypeIO to protect
- * the reset timestamp as well.
- */
- if (i == 0)
- pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
- memset(bktype_shstats, 0, sizeof(*bktype_shstats));
- LWLockRelease(bktype_lock);
+ /* Reset all per-backend entries */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.stats, 0, sizeof(PgStat_BktypeIO));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
}
+
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build IO stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_io_snapshot_cb(void)
{
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- {
- LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
- PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
- PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ PgStat_IO *snap = &pgStatLocal.snapshot.io;
+ dshash_table *hash;
- LWLockAcquire(bktype_lock, LW_SHARED);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
- /*
- * Use the lock in the first BackendType's PgStat_BktypeIO to protect
- * the reset timestamp as well.
- */
- if (i == 0)
- pgStatLocal.snapshot.io.stat_reset_timestamp =
- pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_IO));
- /* using struct assignment due to better type safety */
- *bktype_snap = *bktype_shstats;
- LWLockRelease(bktype_lock);
- }
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_IO, hash, snap);
+
+ LWLockRelease(&shmem->lock);
}
/*
@@ -581,3 +589,116 @@ pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
return true;
}
+
+/*
+ * Accumulate IO counters from src into dst.
+ */
+static inline void
+pgstat_io_accumulate_counters(PgStat_BktypeIO *dst, const PgStat_BktypeIO *src)
+{
+ for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ dst->counts[io_object][io_context][io_op] +=
+ src->counts[io_object][io_context][io_op];
+ dst->bytes[io_object][io_context][io_op] +=
+ src->bytes[io_object][io_context][io_op];
+ dst->times[io_object][io_context][io_op] +=
+ src->times[io_object][io_context][io_op];
+ }
+ }
+ }
+}
+
+/*
+ * Accumulate one per-backend IO entry into a snapshot or the global stats.
+ */
+void
+pgstat_io_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_IO *stats = dst;
+ PgStatShared_IOBackendEntry *e = (PgStatShared_IOBackendEntry *) entry;
+ BackendType bktype = e->header.backend_type;
+
+ if (bktype == B_INVALID)
+ return;
+
+ pgstat_io_accumulate_counters(&stats->stats[bktype], &e->stats.stats);
+}
+
+/*
+ * Accumulate a backend's IO stats into the global stats, then remove the
+ * entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_io_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_IO, &pgStatLocal.shmem->io.lock);
+}
+
+/*
+ * Accumulate all remaining per-backend IO stats entries into the global stats
+ * and remove them. Called at clean server shutdown to ensure all flushed data
+ * is preserved in the stats file.
+ */
+void
+pgstat_io_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_IO, &pgStatLocal.shmem->io.lock);
+}
+
+/*
+ * Returns per-backend IO statistics for the given ProcNumber.
+ */
+PgStat_BackendIO *
+pgstat_fetch_stat_backend_io(ProcNumber procnum)
+{
+ return (PgStat_BackendIO *) pgstat_fetch_per_backend(PGSTAT_KIND_IO, procnum);
+}
+
+/*
+ * Reset a backend's IO stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_io_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ dshash_table *hash;
+ PgStatShared_IOBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ pgstat_io_accumulate_counters(&shmem->stats.stats[entry->header.backend_type],
+ &entry->stats.stats);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats, 0, sizeof(entry->stats));
+ entry->stats.stat_reset_timestamp = ts;
+
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04f2eb21d0b..8080bc1cbba 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -269,7 +269,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* VACUUM command has processed all tables and committed.
*/
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -364,7 +363,6 @@ pgstat_report_analyze(Relation rel,
/* see pgstat_report_vacuum() */
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 3b003f00245..7730dd95766 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -118,12 +118,11 @@ pgstat_dsa_init_size(void)
/*
* The dshash header / initial buckets array needs to fit into "plain"
* shared memory, but it's beneficial to not need dsm segments
- * immediately. A size of 256kB seems works well and is not
- * disproportional compared to other constant sized shared memory
- * allocations. NB: To avoid DSMs further, the user can configure
- * min_dynamic_shared_memory.
+ * immediately. A size of 1MB works well and is not disproportional
+ * compared to other constant sized shared memory allocations. NB: To
+ * avoid DSMs further, the user can configure min_dynamic_shared_memory.
*/
- sz = 256 * 1024;
+ sz = 1024 * 1024;
Assert(dsa_minimum_size() <= sz);
return MAXALIGN(sz);
}
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index ece4d91ed70..99fd20a04c2 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -60,7 +60,6 @@ pgstat_report_wal(bool force)
/* flush IO stats */
pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index c26b50e0d28..d8517abed2e 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1609,32 +1609,47 @@ Datum
pg_stat_get_backend_io(PG_FUNCTION_ARGS)
{
ReturnSetInfo *rsinfo;
- BackendType bktype;
int pid;
- PgStat_Backend *backend_stats;
- PgStat_BktypeIO *bktype_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ BackendType bktype;
+ PgStat_BackendIO *backend_io;
InitMaterializedSRF(fcinfo, 0);
rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, &bktype);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
+ return (Datum) 0;
+
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
return (Datum) 0;
- bktype_stats = &backend_stats->io_stats;
+ bktype = beentry->st_backendType;
+ backend_io = pgstat_fetch_stat_backend_io(procnum);
+
+ if (!backend_io)
+ return (Datum) 0;
/*
* In Assert builds, we can afford an extra loop through all of the
* counters (in pg_stat_io_build_tuples()), checking that only expected
* stats are non-zero, since it keeps the non-Assert code cleaner.
*/
- Assert(pgstat_bktype_io_stats_valid(bktype_stats, bktype));
+ Assert(pgstat_bktype_io_stats_valid(&backend_io->stats, bktype));
+
+ pg_stat_io_build_tuples(rsinfo, &backend_io->stats, bktype,
+ backend_io->stat_reset_timestamp);
- /* save tuples with data from this PgStat_BktypeIO */
- pg_stat_io_build_tuples(rsinfo, bktype_stats, bktype,
- backend_stats->stat_reset_timestamp);
return (Datum) 0;
}
@@ -2115,20 +2130,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
if (!beentry)
PG_RETURN_VOID();
- /* Check if the backend type tracks statistics */
- if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
- PG_RETURN_VOID();
-
/*
- * Accumulate the backend's WAL and lock stats into the global stats, then
- * zero the entries.
+ * Accumulate the backend's WAL, lock and IO stats into the global stats,
+ * then zero the entries.
*/
ts = GetCurrentTimestamp();
pgstat_wal_reset_backend_cb(procNumber, ts);
pgstat_lock_reset_backend_cb(procNumber, ts);
-
- /* Reset IO stats still in PGSTAT_KIND_BACKEND */
- pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
+ pgstat_io_reset_backend_cb(procNumber, ts);
PG_RETURN_VOID();
}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3911ee56943..5d63d19673f 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -218,7 +218,7 @@ typedef struct PgStat_TableXactStatus
* ------------------------------------------------------------
*/
-#define PGSTAT_FILE_FORMAT_ID 0x01A5BCBC
+#define PGSTAT_FILE_FORMAT_ID 0x01A5BCBD
typedef struct PgStat_ArchiverStats
{
@@ -333,6 +333,12 @@ typedef struct PgStat_BktypeIO
PgStat_Counter times[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
} PgStat_BktypeIO;
+typedef struct PgStat_BackendIO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BktypeIO stats;
+} PgStat_BackendIO;
+
typedef struct PgStat_PendingIO
{
uint64 bytes[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
@@ -514,28 +520,6 @@ typedef struct PgStat_WalStats
TimestampTz stat_reset_timestamp;
} PgStat_WalStats;
-/* -------
- * PgStat_Backend Backend statistics
- * -------
- */
-typedef struct PgStat_Backend
-{
- TimestampTz stat_reset_timestamp;
- PgStat_BktypeIO io_stats;
-} PgStat_Backend;
-
-/* ---------
- * PgStat_BackendPending Non-flushed backend stats.
- * ---------
- */
-typedef struct PgStat_BackendPending
-{
- /*
- * Backend statistics store the same amount of IO data as PGSTAT_KIND_IO.
- */
- PgStat_PendingIO pending_io;
-} PgStat_BackendPending;
-
/*
* Functions in pgstat.c
*/
@@ -572,27 +556,6 @@ extern bool pgstat_have_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_report_archiver(const char *xlog, bool failed);
extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void);
-/*
- * Functions in pgstat_backend.c
- */
-
-/* used by pgstat_io.c for I/O stats tracked in backends */
-extern void pgstat_count_backend_io_op_time(IOObject io_object,
- IOContext io_context,
- IOOp io_op,
- instr_time io_time);
-extern void pgstat_count_backend_io_op(IOObject io_object,
- IOContext io_context,
- IOOp io_op, uint32 cnt,
- uint64 bytes);
-
-
-extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber);
-extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid,
- BackendType *bktype);
-extern bool pgstat_tracks_backend_bktype(BackendType bktype);
-extern void pgstat_create_backend(ProcNumber procnum);
-
/*
* Functions in pgstat_bgwriter.c
*/
@@ -623,6 +586,8 @@ extern void pgstat_count_io_op_time(IOObject io_object, IOContext io_context,
uint32 cnt, uint64 bytes);
extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern PgStat_BackendIO *pgstat_fetch_stat_backend_io(ProcNumber procnum);
+extern void pgstat_io_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
extern const char *pgstat_get_io_context_name(IOContext io_context);
extern const char *pgstat_get_io_object_name(IOObject io_object);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9a765eca359..4e4603fe5b1 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -477,12 +477,13 @@ typedef struct PgStatShared_Checkpointer
/* Shared-memory ready PgStat_IO */
typedef struct PgStatShared_IO
{
+ LWLock lock;
+ PgStat_IO stats;
+
/*
- * locks[i] protects stats.stats[i]. locks[0] also protects
- * stats.stat_reset_timestamp.
+ * Per-backend dshash, keyed by ProcNumber.
*/
- LWLock locks[BACKEND_NUM_TYPES];
- PgStat_IO stats;
+ dshash_table_handle backend_hash_handle;
} PgStatShared_IO;
typedef struct PgStatShared_Lock
@@ -513,6 +514,16 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for IO statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_IOBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_BackendIO stats;
+} PgStatShared_IOBackendEntry;
+
/*
* Per-backend entry for lock statistics, stored in a dshash keyed by
* ProcNumber.
@@ -584,12 +595,6 @@ typedef struct PgStatShared_ReplSlot
PgStat_StatReplSlotEntry stats;
} PgStatShared_ReplSlot;
-typedef struct PgStatShared_Backend
-{
- PgStatShared_Common header;
- PgStat_Backend stats;
-} PgStatShared_Backend;
-
/*
* Central shared memory entry for the cumulative stats system.
*
@@ -770,19 +775,6 @@ extern void pgstat_archiver_init_shmem_cb(void *stats);
extern void pgstat_archiver_reset_all_cb(TimestampTz ts);
extern void pgstat_archiver_snapshot_cb(void);
-/*
- * Functions in pgstat_backend.c
- */
-
-/* flags for pgstat_flush_backend() */
-#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO)
-
-extern bool pgstat_flush_backend(bool nowait, uint32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
-extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
- TimestampTz ts);
-
/*
* Functions in pgstat_bgwriter.c
*/
@@ -833,6 +825,9 @@ extern bool pgstat_io_flush_cb(bool nowait);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
+extern void pgstat_io_acc_backend_cb(void);
+extern void pgstat_io_acc_all_backends(void);
+extern void pgstat_io_per_backend_acc_cb(void *dst, void *entry);
/*
* Functions in pgstat_lock.c
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index 2d78a029683..6d53d8e53c6 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -29,16 +29,15 @@
#define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */
#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
-#define PGSTAT_KIND_BACKEND 6 /* per-backend statistics */
/* stats for fixed-numbered objects */
-#define PGSTAT_KIND_ARCHIVER 7
-#define PGSTAT_KIND_BGWRITER 8
-#define PGSTAT_KIND_CHECKPOINTER 9
-#define PGSTAT_KIND_IO 10
-#define PGSTAT_KIND_LOCK 11
-#define PGSTAT_KIND_SLRU 12
-#define PGSTAT_KIND_WAL 13
+#define PGSTAT_KIND_ARCHIVER 6
+#define PGSTAT_KIND_BGWRITER 7
+#define PGSTAT_KIND_CHECKPOINTER 8
+#define PGSTAT_KIND_IO 9
+#define PGSTAT_KIND_LOCK 10
+#define PGSTAT_KIND_SLRU 11
+#define PGSTAT_KIND_WAL 12
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index fae5f939523..e4369221ef3 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -119,15 +119,14 @@ SELECT id, name, fixed_amount,
3 | function | f | f | t
4 | replslot | f | t | t
5 | subscription | f | t | t
- 6 | backend | f | t | f
- 7 | archiver | t | f | t
- 8 | bgwriter | t | f | t
- 9 | checkpointer | t | f | t
- 10 | io | t | f | t
- 11 | lock | t | f | t
- 12 | slru | t | f | t
- 13 | wal | t | f | t
-(13 rows)
+ 6 | archiver | t | f | t
+ 7 | bgwriter | t | f | t
+ 8 | checkpointer | t | f | t
+ 9 | io | t | f | t
+ 10 | lock | t | f | t
+ 11 | slru | t | f | t
+ 12 | wal | t | f | t
+(12 rows)
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;
@@ -1870,13 +1869,20 @@ SELECT :io_stats_post_reset < :io_stats_pre_reset;
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset
FROM pg_stat_get_backend_io(pg_backend_pid()) \gset
--- pg_stat_reset_shared() did not reset backend IO stats
-SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset;
+-- pg_stat_reset_shared() also resets per-backend IO stats
+SELECT :my_io_stats_pre_reset > :my_io_stats_post_reset;
?column?
----------
t
(1 row)
+SELECT bool_and(stats_reset IS NOT NULL) AS backend_io_reset_timestamp_set
+ FROM pg_stat_get_backend_io(pg_backend_pid());
+ backend_io_reset_timestamp_set
+--------------------------------
+ t
+(1 row)
+
-- but pg_stat_reset_backend_stats() does
SELECT pg_stat_reset_backend_stats(pg_backend_pid());
pg_stat_reset_backend_stats
@@ -1903,11 +1909,13 @@ SELECT pg_stat_get_backend_io(0);
------------------------
(0 rows)
--- Auxiliary processes return no data.
-SELECT pg_stat_get_backend_io(:checkpointer_pid);
- pg_stat_get_backend_io
-------------------------
-(0 rows)
+-- Auxiliary processes now return data
+SELECT count(*) > 0 AS checkpointer_has_io_stats
+ FROM pg_stat_get_backend_io(:checkpointer_pid);
+ checkpointer_has_io_stats
+---------------------------
+ t
+(1 row)
-- test BRIN index doesn't block HOT update
CREATE TABLE brin_hot (
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index c17753dcd20..d5877cd8404 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -878,8 +878,10 @@ SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) +
SELECT :io_stats_post_reset < :io_stats_pre_reset;
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset
FROM pg_stat_get_backend_io(pg_backend_pid()) \gset
--- pg_stat_reset_shared() did not reset backend IO stats
-SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset;
+-- pg_stat_reset_shared() also resets per-backend IO stats
+SELECT :my_io_stats_pre_reset > :my_io_stats_post_reset;
+SELECT bool_and(stats_reset IS NOT NULL) AS backend_io_reset_timestamp_set
+ FROM pg_stat_get_backend_io(pg_backend_pid());
-- but pg_stat_reset_backend_stats() does
SELECT pg_stat_reset_backend_stats(pg_backend_pid());
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_backend_reset
@@ -889,8 +891,9 @@ SELECT :my_io_stats_pre_reset > :my_io_stats_post_backend_reset;
-- Check invalid input for pg_stat_get_backend_io()
SELECT pg_stat_get_backend_io(NULL);
SELECT pg_stat_get_backend_io(0);
--- Auxiliary processes return no data.
-SELECT pg_stat_get_backend_io(:checkpointer_pid);
+-- Auxiliary processes now return data
+SELECT count(*) > 0 AS checkpointer_has_io_stats
+ FROM pg_stat_get_backend_io(:checkpointer_pid);
-- test BRIN index doesn't block HOT update
CREATE TABLE brin_hot (
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d89fccf482d..52de6e93c31 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2316,7 +2316,6 @@ PgFdwSamplingMethod
PgFdwScanState
PgIfAddrCallback
PgStatShared_Archiver
-PgStatShared_Backend
PgStatShared_BgWriter
PgStatShared_Checkpointer
PgStatShared_Common
@@ -2326,6 +2325,7 @@ PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
PgStatShared_IO
+PgStatShared_IOBackendEntry
PgStatShared_Lock
PgStatShared_LockBackendEntry
PgStatShared_PerBackendEntry
@@ -2336,8 +2336,7 @@ PgStatShared_Subscription
PgStatShared_Wal
PgStatShared_WalBackendEntry
PgStat_ArchiverStats
-PgStat_Backend
-PgStat_BackendPending
+PgStat_BackendIO
PgStat_BackendSubEntry
PgStat_BgWriterStats
PgStat_BktypeIO
--
2.34.1
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-08-07 01:39 ` Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Michael Paquier @ 2026-08-07 01:39 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Andres Freund <andres@anarazel.de>; Sami Imseih <samimseih@gmail.com>
On Mon, Aug 03, 2026 at 02:20:16PM +0000, Bertrand Drouvot wrote:
> 0001: add tests for per-backend statistics
>
> It adds new tests that will serve as compatibility coverage for the redesign.
> It could be applied while we are discussing the other patches.
That seems useful on its own. Will look at that in details first.
> 0002: add new infrastructure for per-backend statistics
>
> It introduces the common per-backend entry header, the kind metadata needed to
> describe per-backend storage, and backend-local state holding the attached dshash
> and cached entry pointer. It provides the common operations for creating and
> attaching the hashes, creating and caching the current process's entries, fetching
> an individual entry, constructing consistent snapshots, transferring entries into
> global statistics, removing entries, and the accumulation at clean server shutdown.
Okay, so if I get it right when a backend start we get a new reference
to the new dshash you are introducing, then each backend uses this
reference to push its stats updates. This is moving the cost of
aggregating the data when querying the data of each backend for WAL,
IO and lock stats, rather than do twice the aggregate for the central
WAL/IO/lock data plus the backend counterpart on HEAD.
Then the reason why you are using a new dshash to keep track of the
backend data is cost: you need to read all the backend-side data when
querying pg_stat_io, pg_stat_wal or pg_stat_lock, and you don't want
to trigger a full sequential scan of the dshash.
Hmm. First, do we need a dshash at all? The number of backends is
fixed at startup so we could use a set of arrays instead for a cheaper
access (lock, WAL and IO), allocated in shmem? The locking gets fuzzy
then:
- Data is written by each backend.
- Data could be read by anything else.
Using one LWLock for each backend sounds costly just for more
correctness with the stats, and we don't have that many writes anyway?
Second, you may be interested in this patch:
https://www.postgresql.org/message-id/CAA5RZ0supQBxSkh=CWB39=j+cL3hHcLPki3tcBk0B1r4fesg_g@mail.gmail...
This is for PGSS, but could be applied to your patch set. The idea is
simple: keep the stats kind for backends, but register a dedicated
dshash for it rather than having more dsa facilities in the area of
pgstat.c.
pgstat.c becomes much larger, with a bunch of knowledge now related to
backends. Would it make sense to invent a pgstat_snapshot.c where the
existing snapshot-related code now in pgstat.c could be moved? Or
perhaps pgstat_backend.c should be kept, with its snapshot code there?
Keeping all that knowledge in pgstat.c is not really nice, at least
IMV.
> No statistics kind registers per-backend metadata in this patch.
>
> 0003 moves WAL statistics to the infrastructure introduced by 0002.
> 0004 performs the corresponding conversion for lock statistics.
> 0005 performs the corresponding conversion for IO statistics. Once WAL, lock,
> and IO have moved, PGSTAT_KIND_BACKEND contains no data, so this patch also removes
> that kind and pgstat_backend.c, together with their obsolete infrastructure.
0003~0005 are a bit boring, in the good sense. Perhaps you should
split 0005 into a 0005 for the IO move to these new APIs and a 0006 to
remove the stats kind. Based on my other comments, I am not actually
convinced that we absolutely have to drop PGSTAT_KIND_BACKEND, quite
the contrary.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../anU3OE2403PZdwJL@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
@ 2026-08-07 12:31 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 11+ messages in thread
From: Bertrand Drouvot @ 2026-08-07 12:31 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers@lists.postgresql.org, Andres Freund <andres@anarazel.de>; Sami Imseih <samimseih@gmail.com>
Hi,
On Fri, Aug 07, 2026 at 10:39:04AM +0900, Michael Paquier wrote:
> On Mon, Aug 03, 2026 at 02:20:16PM +0000, Bertrand Drouvot wrote:
> > 0001: add tests for per-backend statistics
> >
> > It adds new tests that will serve as compatibility coverage for the redesign.
> > It could be applied while we are discussing the other patches.
>
> That seems useful on its own. Will look at that in details first.
Thanks!
> > 0002: add new infrastructure for per-backend statistics
> >
> Okay, so if I get it right when a backend start we get a new reference
> to the new dshash you are introducing, then each backend uses this
> reference to push its stats updates.
Yeah, that's the idea. Each process creates and caches its entries during
initialization. Flushes then use the cached pointers, without a hash lookup.
Only the global WAL, Lock and I/O queries need to scan and aggregate all live
entries.
> This is moving the cost of
> aggregating the data when querying the data of each backend for WAL,
> IO and lock stats, rather than do twice the aggregate for the central
> WAL/IO/lock data plus the backend counterpart on HEAD.
Right, with one precision: the aggregation cost moves to queries of the global
views, not to queries of an individual backend. A per-backend fetch still looks
up only one ProcNumber entry. On current master, a flush updates both the fixed
global statistics and PGSTAT_KIND_BACKEND. With the new design it updates
only the live entry, and a global query combines the fixed accumulator with all
live entries.
> Then the reason why you are using a new dshash to keep track of the
> backend data is cost: you need to read all the backend-side data when
> querying pg_stat_io, pg_stat_wal or pg_stat_lock, and you don't want
> to trigger a full sequential scan of the dshash.
>
> Hmm. First, do we need a dshash at all? The number of backends is
> fixed at startup so we could use a set of arrays instead for a cheaper
> access (lock, WAL and IO), allocated in shmem?
Yeah, that was my first comment in the "Design explanation for the new hashes"
section of my first email in this thread:
an earlier POC version used arrays indexed by ProcNumber. The main concern was
reserving storage for every possible process slot and every kind, even with
few active processes. With that, max_connections=10000 would reserve about 33.5MB
for the three kinds. Queries would also need to scan unused slots or use another
structure to track active ones.
I don't think that the fixed allocation would be the right design, particularly
because it grows with MaxBackends even when most slots are unused. Since v1
already avoids hash lookups on the flush path, I don't think the simpler access
justifies that memory cost.
> Using one LWLock for each backend sounds costly just for more
> correctness with the stats, and we don't have that many writes anyway?
I think that an array would not remove the need for content synchronization. The
dshash partition lock protects the entry lifetime, but the owner updates its
cached entry without holding that lock. Queries read the entry, while shared
and per-backend resets can modify it from another process. In particular, a
concurrent reset with a flush could lose counters.
v1 uses one LWLock per kind and live process for that.
> Second, you may be interested in this patch:
> https://www.postgresql.org/message-id/CAA5RZ0supQBxSkh=CWB39=j+cL3hHcLPki3tcBk0B1r4fesg_g@mail.gmail...
> This is for PGSS, but could be applied to your patch set. The idea is
> simple: keep the stats kind for backends, but register a dedicated
> dshash for it rather than having more dsa facilities in the area of
> pgstat.c.
Yeah, I started looking at Sami's patch yesterday, after you mentioned it to me
off-list.
PGSTAT_KIND_BACKEND is variable-numbered, so it could directly use own_hash.
Also, keeping PGSTAT_KIND_BACKEND and using own_hash could still eliminate duplicate
accounting if routine flushes updated only its live entry, while the fixed WAL,
Lock and I/O structures retained on exit transferred statistics.
That said, I don't think keeping the combined kind is a good fit for this design:
A WAL snapshot or reset would acquire the same entry lock used by Lock and I/O
flushes. Exit and ProcNumber reuse would also have to coordinate one combined
entry with the three fixed accumulators and their locks.
In the new design, each fixed kind owns both parts of its statistics: its fixed
accumulator and its live per-process entries. A snapshot, reset, or transfer
therefore involves only that kind's lock and hash.
IIUC, own_hash would only change where the generic variable-statistics entries
are stored: it would retain their refcount, drop and garbage collection machinery.
V1 instead uses ProcNumber keyed entries with a process lifetime lifecycle, so
that machinery is not needed.
One detail is that v1 does not add another DSA: all three hashes share the
existing pgstat DSA, while Sami's patch creates a dedicated DSA for an own_hash
kind.
So, while own_hash could be used to implement a combined one hash alternative,
I don't think it provides the same per-kind isolation or ownership as v1.
> pgstat.c becomes much larger, with a bunch of knowledge now related to
> backends.
Agreed. I wonder if introducing pgstat_per_backend.c wouldn't make more sense.
The added code handles entry creation, fetching, transfer and removal in addition
to snapshots, so pgstat_snapshot.c seems too narrow.
> 0003~0005 are a bit boring, in the good sense. Perhaps you should
> split 0005 into a 0005 for the IO move to these new APIs and a 0006 to
> remove the stats kind.
Yeah, good point. Moving the removal into 0006 would make more sense.
> Based on my other comments, I am not actually
> convinced that we absolutely have to drop PGSTAT_KIND_BACKEND, quite
> the contrary.
Right, as mentioned above, it could be retained, but I don't think it should be.
It would keep WAL, Lock and I/O coupled through one entry and content lock, while
exit and ProcNumber reuse would need to coordinate that entry with three fixed
accumulators. Every additional per-backend statistic would then add more contention.
In the proposed design, each kind owns both its live entries and transferred
statistics.
Adding another per-backend statistic does not make it share an entry or content
lock with existing kinds.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-08-10 04:38 ` Michael Paquier <michael@paquier.xyz>
2026-08-10 14:45 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Michael Paquier @ 2026-08-10 04:38 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Andres Freund <andres@anarazel.de>; Sami Imseih <samimseih@gmail.com>
On Fri, Aug 07, 2026 at 12:31:47PM +0000, Bertrand Drouvot wrote:
> On Fri, Aug 07, 2026 at 10:39:04AM +0900, Michael Paquier wrote:
> an earlier POC version used arrays indexed by ProcNumber. The main concern was
> reserving storage for every possible process slot and every kind, even with
> few active processes. With that, max_connections=10000 would reserve about 33.5MB
> for the three kinds. Queries would also need to scan unused slots or use another
> structure to track active ones.
Hmm, okay, that feels like a waste. I doubt that anybody would use
that many max_connections, but if these are idle most of the time..
> I don't think that the fixed allocation would be the right design, particularly
> because it grows with MaxBackends even when most slots are unused. Since v1
> already avoids hash lookups on the flush path, I don't think the simpler access
> justifies that memory cost.
Accessing an array indexed by procnumber should be slightly cheaper
than a hash lookup when grabbing the stats of an individual backend,
as this is just a BackendPidGetProc() -> GetNumberFromPGProc() to get
a location.
>> Using one LWLock for each backend sounds costly just for more
>> correctness with the stats, and we don't have that many writes anyway?
>
> I think that an array would not remove the need for content
> synchronization.
We would need some level of concurrent protection even with an array.
> The dshash partition lock protects the entry lifetime, but the owner updates its
> cached entry without holding that lock. Queries read the entry, while shared
> and per-backend resets can modify it from another process. In particular, a
> concurrent reset with a flush could lose counters.
>
> v1 uses one LWLock per kind and live process for that.
I can see that:
+pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash, void *snap)
[...]
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->lock, LW_SHARED);
That's a sequential scan combined with potentially hundreds of LWLocks
acquired and released successivelly. That looks expensive here for a
single IO/lock/WAL data scan. That's the level of locking required
because a mutex cannot be hold while doing external calls, and here we
have one per_backend_acc_cb callback and one
pgstat_cache_per_backend_entry(). Not sure I like much this costly
locking level. I'm concerned by this cost.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../anlVx_TD_Q8He5uf@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
@ 2026-08-10 14:45 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-13 15:41 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Bertrand Drouvot @ 2026-08-10 14:45 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers@lists.postgresql.org, Andres Freund <andres@anarazel.de>; Sami Imseih <samimseih@gmail.com>
Hi,
On Mon, Aug 10, 2026 at 01:38:31PM +0900, Michael Paquier wrote:
> Accessing an array indexed by procnumber should be slightly cheaper
> than a hash lookup when grabbing the stats of an individual backend,
> as this is just a BackendPidGetProc() -> GetNumberFromPGProc() to get
> a location.
Right, an array would make fetching an individual backend slightly cheaper.
My point was only that routine flushes use cached entry pointers and therefore
avoid hash lookups.
> I can see that:
>
> +pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash, void *snap)
> [...]
> + while ((entry = dshash_seq_next(&hstat)) != NULL)
> + {
> + LWLockAcquire(&entry->lock, LW_SHARED);
>
> That's a sequential scan combined with potentially hundreds of LWLocks
> acquired and released successivelly. That looks expensive here for a
> single IO/lock/WAL data scan. That's the level of locking required
> because a mutex cannot be hold while doing external calls, and here we
> have one per_backend_acc_cb callback and one
> pgstat_cache_per_backend_entry(). Not sure I like much this costly
> locking level. I'm concerned by this cost.
Yeah, I benchmarked this against unpatched master (-O2 and assertions disabled).
Each sessions generated and flushed WAL/IO statistics, then remained connected
and idle. Then queried pg_stat_wal, pg_stat_lock and pg_stat_io:
Mean latency in ms:
400 backends 10000 backends
master v1 master v1
WAL 0.017 0.029 0.017 0.354
Lock 0.018 0.033 0.018 0.575
IO 0.069 0.143 0.069 2.847
Those are warmed, continuously repeated queries.
Now the impact:
- the extra timing is only when querying the corresponding global view
- the shared entry lock conflicts only with exclusive operations on the same
backend's entry for that statistics kind. The usual statistics flush uses
LWLockConditionalAcquire(), so it leaves counters pending rather than waiting.
Forced flushes, resets, and backend exit processing may wait, but other backends
can continue flushing their own entries.
FWIW, this kind of scan is not new:
- pg_locks walks the PGPROC slots and takes each live process's fpInfoLock in
shared mode.
- pg_stat_activity also performs a scan, although it uses a lockless changecount
and retry protocol rather than taking one LWLock per backend.
- the current statistics implementation with stats_fetch_consistency = snapshot
also scans the shared statistics dshash and takes each entry's content LWLock in
shared mode while copying it.
For comparison, select count(*) FROM pg_stat_activity took about 57 ms and select
count(*) FROM pg_locks took about 4ms, both with the same 10000 connections.
Given that the cost is still sub millisecond at hundreds of connections and a few
milliseconds at 10000 and given the impact mentioned above, I don't think this is
a practical blocker though. What do you think?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-10 14:45 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-08-13 15:41 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 07:15 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Bertrand Drouvot @ 2026-08-13 15:41 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers@lists.postgresql.org, Andres Freund <andres@anarazel.de>; Sami Imseih <samimseih@gmail.com>
Hi,
On Mon, Aug 10, 2026 at 02:45:16PM +0000, Bertrand Drouvot wrote:
> Given that the cost is still sub millisecond at hundreds of connections and a few
> milliseconds at 10000 and given the impact mentioned above, I don't think this is
> a practical blocker though. What do you think?
>
A rebase was needed, so while at it, I implemented what we have discussed so far,
means:
- moved the generic per-backend statistics infrastructure to a new pgstat_per_backend.c
file
- split the previous 0005 into the IO migration and the removal of PGSTAT_KIND_BACKEND
and pgstat_backend.c.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v2-0001-pgstat-add-tests-for-per-backend-statistics.patch (21.2K, ../../an3lxtz4TaYXXg%2F8@bdtpg/2-v2-0001-pgstat-add-tests-for-per-backend-statistics.patch)
download | inline diff:
From 997f287af3697b03011afc95949491bc0e8d1b73 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:23:40 +0000
Subject: [PATCH v2 1/6] pgstat: add tests for per-backend statistics
Add an isolation test for per-backend WAL statistics. Verify that SNAPSHOT, CACHE
and NONE modes behave as expected.
Add a TAP test that starts a backend after another session has built a
statistics snapshot. Per-backend WAL, IO, and Lock queries for the new
backend must return no rows, rather than fabricated rows containing zeroes.
Extend test_shm_mq with a shared-memory-only worker that reports WAL usage
through a routine nonblocking flush and remains alive while the controlling
backend verifies that the global WAL counters include it.
Finally, extend the core statistics regression test for per-backend WAL
reset. Check that resetting one backend reduces its counters and sets its reset
timestamp without removing the same activity from the global WAL statistics.
These tests will serve as compatibility coverage for the per-backend redesign
changes coming in following commits.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
.../isolation/expected/stats-per-backend.out | 143 ++++++++++++++++++
src/test/isolation/isolation_schedule | 1 +
.../isolation/specs/stats-per-backend.spec | 123 +++++++++++++++
src/test/modules/test_misc/meson.build | 1 +
.../modules/test_misc/t/015_stats_snapshot.pl | 48 ++++++
.../test_shm_mq/expected/test_shm_mq.out | 7 +
.../modules/test_shm_mq/sql/test_shm_mq.sql | 3 +
src/test/modules/test_shm_mq/test.c | 51 +++++++
.../modules/test_shm_mq/test_shm_mq--1.0.sql | 4 +
src/test/modules/test_shm_mq/test_shm_mq.h | 4 +
src/test/modules/test_shm_mq/worker.c | 10 ++
src/test/regress/expected/stats.out | 29 ++++
src/test/regress/sql/stats.sql | 10 ++
13 files changed, 434 insertions(+)
30.6% src/test/isolation/expected/
31.4% src/test/isolation/specs/
9.7% src/test/modules/test_misc/t/
17.6% src/test/modules/test_shm_mq/
5.6% src/test/regress/expected/
4.3% src/test/regress/sql/
diff --git a/src/test/isolation/expected/stats-per-backend.out b/src/test/isolation/expected/stats-per-backend.out
new file mode 100644
index 00000000000..0d85ae99033
--- /dev/null
+++ b/src/test/isolation/expected/stats-per-backend.out
@@ -0,0 +1,143 @@
+Parsed test spec with 2 sessions
+
+starting permutation: s1_fetch_consistency_snapshot s1_begin s1_build_snapshot s2_generate s1_check_snapshot_backend s1_commit s1_check_live_backend
+step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot';
+step s1_begin: BEGIN;
+step s1_build_snapshot:
+ SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal;
+
+snapshot_created
+----------------
+t
+(1 row)
+
+step s2_generate:
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_check_snapshot_backend:
+ SELECT wal_records = 0 AS snapshot_excludes_later_update
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+snapshot_excludes_later_update
+------------------------------
+t
+(1 row)
+
+step s1_commit: COMMIT;
+step s1_check_live_backend:
+ SELECT wal_records > 0 AS live_entry_exists
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+live_entry_exists
+-----------------
+t
+(1 row)
+
+
+starting permutation: s1_fetch_consistency_cache s2_generate s1_begin s1_save_backend_stats s2_generate_more s1_check_cached_backend s1_commit s1_check_refreshed_backend
+step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache';
+step s2_generate:
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_begin: BEGIN;
+step s1_save_backend_stats:
+ INSERT INTO stats_per_backend_saved
+ SELECT wal_records
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+step s2_generate_more:
+ INSERT INTO stats_per_backend_data VALUES (2);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_check_cached_backend:
+ SELECT current.wal_records = saved.wal_records AS cache_is_stable
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+
+cache_is_stable
+---------------
+t
+(1 row)
+
+step s1_commit: COMMIT;
+step s1_check_refreshed_backend:
+ SELECT current.wal_records > saved.wal_records AS cache_is_refreshed
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+
+cache_is_refreshed
+------------------
+t
+(1 row)
+
+
+starting permutation: s1_reset_backend s1_check_backend_reset
+step s1_reset_backend:
+ SELECT pg_stat_reset_backend_stats(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+pg_stat_reset_backend_stats
+---------------------------
+
+(1 row)
+
+step s1_check_backend_reset:
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+reset_timestamp_set
+-------------------
+t
+(1 row)
+
+
+starting permutation: s1_reset_shared s1_check_shared_reset
+step s1_reset_shared:
+ SELECT pg_stat_reset_shared('wal');
+
+pg_stat_reset_shared
+--------------------
+
+(1 row)
+
+step s1_check_shared_reset:
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+reset_timestamp_set
+-------------------
+t
+(1 row)
+
diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule
index a27480a86a2..44eb2edc07f 100644
--- a/src/test/isolation/isolation_schedule
+++ b/src/test/isolation/isolation_schedule
@@ -105,6 +105,7 @@ test: vacuum-concurrent-drop
test: vacuum-conflict
test: vacuum-skip-locked
test: stats
+test: stats-per-backend
test: horizons
test: predicate-hash
test: predicate-gist
diff --git a/src/test/isolation/specs/stats-per-backend.spec b/src/test/isolation/specs/stats-per-backend.spec
new file mode 100644
index 00000000000..5dfc8dbe90e
--- /dev/null
+++ b/src/test/isolation/specs/stats-per-backend.spec
@@ -0,0 +1,123 @@
+# Test snapshot, cache, and reset behavior for per-backend statistics.
+#
+# Session s2 generates and flushes WAL records, which provide a deterministic
+# per-backend counter. Session s1 verifies that SNAPSHOT mode fixes all
+# per-backend entries at the initial statistics snapshot, while CACHE mode
+# fixes each entry on first access and refreshes it after the transaction
+# ends. The final permutations verify reset timestamps after per-backend and
+# shared resets.
+
+setup
+{
+ CREATE TABLE stats_per_backend_data(id int);
+ CREATE TABLE stats_per_backend_saved(wal_records bigint);
+}
+
+teardown
+{
+ DROP TABLE stats_per_backend_data;
+ DROP TABLE stats_per_backend_saved;
+}
+
+session s1
+setup { SET stats_fetch_consistency = 'none'; }
+
+step s1_fetch_consistency_cache { SET stats_fetch_consistency = 'cache'; }
+step s1_fetch_consistency_snapshot { SET stats_fetch_consistency = 'snapshot'; }
+step s1_begin { BEGIN; }
+step s1_commit { COMMIT; }
+step s1_build_snapshot {
+ SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal;
+}
+step s1_check_snapshot_backend {
+ SELECT wal_records = 0 AS snapshot_excludes_later_update
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_live_backend {
+ SELECT wal_records > 0 AS live_entry_exists
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_save_backend_stats {
+ INSERT INTO stats_per_backend_saved
+ SELECT wal_records
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_cached_backend {
+ SELECT current.wal_records = saved.wal_records AS cache_is_stable
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+}
+step s1_check_refreshed_backend {
+ SELECT current.wal_records > saved.wal_records AS cache_is_refreshed
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+}
+step s1_reset_backend {
+ SELECT pg_stat_reset_backend_stats(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_backend_reset {
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_reset_shared {
+ SELECT pg_stat_reset_shared('wal');
+}
+step s1_check_shared_reset {
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+
+session s2
+step s2_generate {
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+}
+step s2_generate_more {
+ INSERT INTO stats_per_backend_data VALUES (2);
+ SELECT pg_stat_force_next_flush();
+}
+
+# with stats_fetch_consistency=snapshot s1 should not see flushed changes from
+# s2 after building the statistics snapshot, but should see them after commit
+permutation
+ s1_fetch_consistency_snapshot
+ s1_begin
+ s1_build_snapshot
+ s2_generate
+ s1_check_snapshot_backend
+ s1_commit
+ s1_check_live_backend
+
+# with stats_fetch_consistency=cache s1 should not see flushed changes from s2
+# after the first access, but should see them after commit
+permutation
+ s1_fetch_consistency_cache
+ s2_generate
+ s1_begin
+ s1_save_backend_stats
+ s2_generate_more
+ s1_check_cached_backend
+ s1_commit
+ s1_check_refreshed_backend
+
+# a per-backend reset should set that backend's reset timestamp
+permutation s1_reset_backend s1_check_backend_reset
+
+# a shared WAL reset should set the reset timestamp for live backends
+permutation s1_reset_shared s1_check_shared_reset
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index ee290698b31..cf4c8558f3d 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -23,6 +23,7 @@ tests += {
't/012_ddlutils.pl',
't/013_temp_obj_multisession.pl',
't/014_log_statement_max_length.pl',
+ 't/015_stats_snapshot.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/015_stats_snapshot.pl b/src/test/modules/test_misc/t/015_stats_snapshot.pl
new file mode 100644
index 00000000000..43c9157bbb4
--- /dev/null
+++ b/src/test/modules/test_misc/t/015_stats_snapshot.pl
@@ -0,0 +1,48 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Verify that a statistics snapshot excludes backends that started after the
+# snapshot was created.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('stats_snapshot');
+$node->init;
+$node->start;
+
+my $snapshot = $node->background_psql('postgres');
+
+$snapshot->query_safe(
+ q[
+ BEGIN;
+ SET LOCAL stats_fetch_consistency = 'snapshot';
+ SELECT count(*) FROM pg_stat_wal;
+]);
+
+# Establish this connection only after the first session built its snapshot.
+my $late = $node->background_psql('postgres');
+my $late_pid = $late->query_safe('SELECT pg_backend_pid()');
+
+# WAL returns one scalar composite, so count a field to distinguish a NULL
+# result from the fabricated all-zero row.
+is( $snapshot->query_safe(
+ qq[
+ SELECT
+ (SELECT count(wal_records)
+ FROM pg_stat_get_backend_wal($late_pid)),
+ (SELECT count(*) FROM pg_stat_get_backend_io($late_pid)),
+ (SELECT count(*) FROM pg_stat_get_backend_lock($late_pid));
+ ]),
+ '0|0|0',
+ 'statistics snapshot excludes a backend created later');
+
+$snapshot->query_safe('ROLLBACK');
+$late->quit;
+$snapshot->quit;
+$node->stop;
+
+done_testing();
diff --git a/src/test/modules/test_shm_mq/expected/test_shm_mq.out b/src/test/modules/test_shm_mq/expected/test_shm_mq.out
index c4858b0c205..ec3a7d3199e 100644
--- a/src/test/modules/test_shm_mq/expected/test_shm_mq.out
+++ b/src/test/modules/test_shm_mq/expected/test_shm_mq.out
@@ -34,3 +34,10 @@ SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int
(1 row)
+-- A shmem-only worker can publish statistics before its clean exit.
+SELECT test_shm_mq_worker_stats();
+ test_shm_mq_worker_stats
+--------------------------
+ t
+(1 row)
+
diff --git a/src/test/modules/test_shm_mq/sql/test_shm_mq.sql b/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
index 9de19d304a2..e80e4399881 100644
--- a/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
+++ b/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
@@ -10,3 +10,6 @@ SELECT test_shm_mq(1024, 'a', 2001, 1);
SELECT test_shm_mq(32768, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,(100+900*random())::int)), 10000, 1);
SELECT test_shm_mq(100, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,(100+200*random())::int)), 10000, 1);
SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,270000)), 200, 3);
+
+-- A shmem-only worker can publish statistics before its clean exit.
+SELECT test_shm_mq_worker_stats();
diff --git a/src/test/modules/test_shm_mq/test.c b/src/test/modules/test_shm_mq/test.c
index 0e55287e510..4464d796ef9 100644
--- a/src/test/modules/test_shm_mq/test.c
+++ b/src/test/modules/test_shm_mq/test.c
@@ -26,6 +26,7 @@ PG_MODULE_MAGIC;
PG_FUNCTION_INFO_V1(test_shm_mq);
PG_FUNCTION_INFO_V1(test_shm_mq_pipelined);
+PG_FUNCTION_INFO_V1(test_shm_mq_worker_stats);
static void verify_message(Size origlen, char *origdata, Size newlen,
char *newdata);
@@ -253,6 +254,56 @@ test_shm_mq_pipelined(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+/*
+ * Verify that a shmem-only worker can publish statistics through a routine
+ * nonblocking flush while it remains alive.
+ */
+Datum
+test_shm_mq_worker_stats(PG_FUNCTION_ARGS)
+{
+ const char *message = PG_TEST_SHM_MQ_STATS_MESSAGE;
+ PgStat_Counter before;
+ PgStat_Counter after;
+ dsm_segment *seg;
+ shm_mq_handle *outqh;
+ shm_mq_handle *inqh;
+ shm_mq_result res;
+ Size len;
+ void *data;
+
+ pgstat_clear_snapshot();
+ before = pgstat_fetch_stat_wal()->wal_counters.wal_records;
+
+ test_shm_mq_setup(1024, 1, &seg, &outqh, &inqh);
+
+ res = shm_mq_send(outqh, strlen(message), message, false, true);
+
+ if (res != SHM_MQ_SUCCESS)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not send statistics test message")));
+
+ res = shm_mq_receive(inqh, &len, &data, false);
+
+ if (res != SHM_MQ_SUCCESS)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not receive statistics test message")));
+
+ verify_message(strlen(message), (char *) message, len, data);
+
+ /*
+ * The worker is waiting for another message here. Check its live entry
+ * before detaching the queues and triggering its shutdown flush.
+ */
+ pgstat_clear_snapshot();
+ after = pgstat_fetch_stat_wal()->wal_counters.wal_records;
+
+ dsm_detach(seg);
+
+ PG_RETURN_BOOL(after >= before + PG_TEST_SHM_MQ_STATS_RECORDS);
+}
+
/*
* Verify that two messages are the same.
*/
diff --git a/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql b/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
index 56db05d93df..dce6f7378a5 100644
--- a/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
+++ b/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
@@ -17,3 +17,7 @@ CREATE FUNCTION test_shm_mq_pipelined(queue_size pg_catalog.int8,
verify pg_catalog.bool default true)
RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION test_shm_mq_worker_stats()
+ RETURNS pg_catalog.bool
+ AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_shm_mq/test_shm_mq.h b/src/test/modules/test_shm_mq/test_shm_mq.h
index b6a0290289c..501be180642 100644
--- a/src/test/modules/test_shm_mq/test_shm_mq.h
+++ b/src/test/modules/test_shm_mq/test_shm_mq.h
@@ -21,6 +21,10 @@
/* Identifier for shared memory segments used by this extension. */
#define PG_TEST_SHM_MQ_MAGIC 0x79fb2447
+/* Special message used to test statistics from a shmem-only worker. */
+#define PG_TEST_SHM_MQ_STATS_MESSAGE "test_shm_mq worker stats"
+#define PG_TEST_SHM_MQ_STATS_RECORDS 1000000
+
/*
* This structure is stored in the dynamic shared memory segment. We use
* it to determine whether all workers started up OK and successfully
diff --git a/src/test/modules/test_shm_mq/worker.c b/src/test/modules/test_shm_mq/worker.c
index 0ba1cfcac47..afcea7fc0cb 100644
--- a/src/test/modules/test_shm_mq/worker.c
+++ b/src/test/modules/test_shm_mq/worker.c
@@ -19,6 +19,7 @@
#include "postgres.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -27,6 +28,7 @@
#include "storage/shm_mq.h"
#include "storage/shm_toc.h"
#include "tcop/tcopprot.h"
+#include "utils/pgstat_internal.h"
#include "test_shm_mq.h"
@@ -185,6 +187,14 @@ copy_messages(shm_mq_handle *inqh, shm_mq_handle *outqh)
if (res != SHM_MQ_SUCCESS)
break;
+ if (len == strlen(PG_TEST_SHM_MQ_STATS_MESSAGE) &&
+ memcmp(data, PG_TEST_SHM_MQ_STATS_MESSAGE, len) == 0)
+ {
+ pgWalUsage.wal_records += PG_TEST_SHM_MQ_STATS_RECORDS;
+ pgstat_report_fixed = true;
+ (void) pgstat_report_stat(false);
+ }
+
/* Send it back out. */
res = shm_mq_send(outqh, len, data, false, true);
if (res != SHM_MQ_SUCCESS)
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index c682a9ed60a..9db59e25210 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1045,6 +1045,35 @@ SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_bac
t
(1 row)
+-- Test pg_stat_reset_backend_stats() for WAL
+SELECT wal_bytes AS wal_bytes_before_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT pg_stat_reset_backend_stats(pg_backend_pid());
+ pg_stat_reset_backend_stats
+-----------------------------
+
+(1 row)
+
+SELECT wal_bytes AS wal_bytes_after_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT :wal_bytes_after_reset < :wal_bytes_before_reset;
+ ?column?
+----------
+ t
+(1 row)
+
+-- global view should still have the data
+SELECT wal_bytes >= :wal_bytes_before_reset FROM pg_stat_wal;
+ ?column?
+----------
+ t
+(1 row)
+
+-- reset timestamp should be set
+SELECT stats_reset IS NOT NULL FROM pg_stat_get_backend_wal(pg_backend_pid());
+ ?column?
+----------
+ t
+(1 row)
+
-- Test pg_stat_get_backend_idset() and some allied functions.
-- In particular, verify that their notion of backend ID matches
-- our temp schema index.
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index d4623c32cd3..176bd008673 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -482,6 +482,16 @@ SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal;
SELECT pg_stat_force_next_flush();
SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_backend_pid());
+-- Test pg_stat_reset_backend_stats() for WAL
+SELECT wal_bytes AS wal_bytes_before_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT pg_stat_reset_backend_stats(pg_backend_pid());
+SELECT wal_bytes AS wal_bytes_after_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT :wal_bytes_after_reset < :wal_bytes_before_reset;
+-- global view should still have the data
+SELECT wal_bytes >= :wal_bytes_before_reset FROM pg_stat_wal;
+-- reset timestamp should be set
+SELECT stats_reset IS NOT NULL FROM pg_stat_get_backend_wal(pg_backend_pid());
+
-- Test pg_stat_get_backend_idset() and some allied functions.
-- In particular, verify that their notion of backend ID matches
-- our temp schema index.
--
2.34.1
[text/x-diff] v2-0002-pgstat-add-new-infrastructure-for-per-backend-sta.patch (25.4K, ../../an3lxtz4TaYXXg%2F8@bdtpg/3-v2-0002-pgstat-add-new-infrastructure-for-per-backend-sta.patch)
download | inline diff:
From 574d1fbf740e706b5b80895c3af0a70e426c27d8 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 29 Jul 2026 14:11:27 +0000
Subject: [PATCH v2 2/6] pgstat: add new infrastructure for per-backend
statistics
Add a new infrastructure for per-backend statistics that keep one shared entry
per live backend while keeping their existing shared data as global stats for
exited backends.
Define a common dshash entry header containing the ProcNumber key, BackendType,
and content LWLock. Extend PgStat_KindInfo with per-backend statistics related
informations.
At shared-memory initialization, create a dshash for every participating kind.
Create entries during pgstat_initialize(), so that auxiliary and shared memory
only processes are covered. Nonblocking flushes can then acquire the cached
entry's content lock without performing a dshash lookup, DSA address resolution,
allocation, or hash resize.
Add helpers to fetch per-backend entries, include live entries in global
snapshots, and transfer entries to the global statistics when backends exit.
Add a local cache keyed by statistics kind and ProcNumber for per-backend
fetches.
No statistics kind registers per backend metadata in this commit, so the new
shared memory creation and backend initialization loops are no ops. Subsequent
commits will add WAL, Lock, and IO statistics into the infrastructure individually.
Limit this infrastructure to built in fixed numbered statistics kinds as this is
the only use case we have had so far. We could extend to variable ones later on
if needed.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 21 +
.../utils/activity/pgstat_per_backend.c | 513 ++++++++++++++++++
src/backend/utils/activity/pgstat_shmem.c | 46 ++
src/include/utils/pgstat_internal.h | 45 ++
src/tools/pgindent/typedefs.list | 4 +
7 files changed, 631 insertions(+)
89.6% src/backend/utils/activity/
9.7% src/include/utils/
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index 2e32d1485d6..8924b10ed2e 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -29,6 +29,7 @@ OBJS = \
pgstat_io.o \
pgstat_kind.o \
pgstat_lock.o \
+ pgstat_per_backend.o \
pgstat_relation.o \
pgstat_replslot.o \
pgstat_shmem.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index e6dcb2e26fc..6434c377e53 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -14,6 +14,7 @@ backend_sources += files(
'pgstat_io.c',
'pgstat_kind.c',
'pgstat_lock.c',
+ 'pgstat_per_backend.c',
'pgstat_relation.c',
'pgstat_replslot.c',
'pgstat_shmem.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index fe1637e2095..8e580fa7b89 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -70,6 +70,7 @@
* To keep things manageable, stats handling is split across several
* files. Infrastructure pieces are in:
* - pgstat.c - this file, to tie it all together
+ * - pgstat_per_backend.c - generic per-backend statistics infrastructure
* - pgstat_shmem.c - nearly everything dealing with shared memory, including
* the maintenance of hashtable entries
* - pgstat_xact.c - transactional integration, including the transactional
@@ -689,6 +690,13 @@ pgstat_initialize(void)
pgstat_attach_shmem();
+ /*
+ * Create and cache per-backend statistics entries here. This also covers
+ * processes that never call InitPostgres(), such as shared-memory-only
+ * background workers.
+ */
+ pgstat_create_my_per_backend_entries();
+
pgstat_init_snapshot_fixed();
/* Backend initialization callbacks */
@@ -962,6 +970,7 @@ pgstat_clear_snapshot(void)
/* Reset variables */
pgStatLocal.snapshot.context = NULL;
+ pgStatLocal.snapshot.per_backend_stats = NULL;
}
/*
@@ -975,6 +984,13 @@ pgstat_clear_snapshot(void)
force_stats_snapshot_clear = false;
}
+void
+pgstat_maybe_clear_snapshot(void)
+{
+ if (force_stats_snapshot_clear)
+ pgstat_clear_snapshot();
+}
+
void *
pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *may_free)
{
@@ -1556,6 +1572,11 @@ pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
errdetail("Custom cumulative statistics must be registered while initializing modules in \"%s\".",
"shared_preload_libraries")));
+ if (kind_info->per_backend_data_len != 0)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Per-backend statistics are not supported for custom cumulative statistics.")));
+
/*
* Check some data for fixed-numbered stats.
*/
diff --git a/src/backend/utils/activity/pgstat_per_backend.c b/src/backend/utils/activity/pgstat_per_backend.c
new file mode 100644
index 00000000000..b82787e9ea5
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_per_backend.c
@@ -0,0 +1,513 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_per_backend.c
+ * Generic infrastructure for per-backend statistics.
+ *
+ * This file manages the dedicated per-kind dshashes used for per-backend
+ * statistics, including entry creation, fetching, snapshots, transfer to
+ * global statistics, and removal.
+ *
+ * Copyright (c) 2001-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_per_backend.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/dshash.h"
+#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+
+/* hash table for per-backend stats snapshot entries */
+typedef struct PgStat_PerBackendSnapshotKey
+{
+ PgStat_Kind kind;
+ ProcNumber procnum;
+} PgStat_PerBackendSnapshotKey;
+
+typedef struct PgStat_PerBackendSnapshotEntry
+{
+ PgStat_PerBackendSnapshotKey key;
+ char status; /* for simplehash use */
+ void *data; /* the stats data itself */
+} PgStat_PerBackendSnapshotEntry;
+
+#define SH_PREFIX pgstat_per_backend_snapshot
+#define SH_ELEMENT_TYPE PgStat_PerBackendSnapshotEntry
+#define SH_KEY_TYPE PgStat_PerBackendSnapshotKey
+#define SH_KEY key
+#define SH_HASH_KEY(tb, key) \
+ fasthash32((const char *) &key, sizeof(PgStat_PerBackendSnapshotKey), 0)
+#define SH_EQUAL(tb, a, b) \
+ (memcmp(&a, &b, sizeof(PgStat_PerBackendSnapshotKey)) == 0)
+#define SH_SCOPE static inline
+#define SH_DEFINE
+#define SH_DECLARE
+#include "lib/simplehash.h"
+
+/* Per-kind, backend-local state for the per-backend dshashes. */
+typedef struct PgStat_PerBackendLocalState
+{
+ dshash_table *hash;
+ PgStatShared_PerBackendEntry *my_entry;
+} PgStat_PerBackendLocalState;
+
+static PgStat_PerBackendLocalState per_backend_states[PGSTAT_KIND_BUILTIN_SIZE];
+
+/*
+ * Look up a per-backend stats entry in the backend snapshot hash.
+ *
+ * The returned entry may be empty when no matching statistics were found on
+ * first access.
+ */
+static PgStat_PerBackendSnapshotEntry *
+pgstat_lookup_per_backend_entry(PgStat_Kind kind, ProcNumber procnum)
+{
+ PgStat_PerBackendSnapshotKey key;
+
+ if (pgStatLocal.snapshot.per_backend_stats == NULL)
+ return NULL;
+
+ key.kind = kind;
+ key.procnum = procnum;
+
+ return pgstat_per_backend_snapshot_lookup(pgStatLocal.snapshot.per_backend_stats,
+ key);
+}
+
+/*
+ * Cache a per-backend stats entry in the backend snapshot hash.
+ *
+ * If data is NULL, cache an empty entry to record that no matching statistics
+ * were found on first access.
+ */
+static void *
+pgstat_cache_per_backend_entry(PgStat_Kind kind, ProcNumber procnum,
+ const void *data)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendSnapshotKey key;
+ PgStat_PerBackendSnapshotEntry *entry;
+ bool found;
+
+ Assert(pgstat_fetch_consistency > PGSTAT_FETCH_CONSISTENCY_NONE);
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ /* Ensure snapshot context exists */
+ if (!pgStatLocal.snapshot.context)
+ pgStatLocal.snapshot.context = AllocSetContextCreate(TopMemoryContext,
+ "PgStat Snapshot",
+ ALLOCSET_SMALL_SIZES);
+
+ /* Create per-backend hash on first use */
+ if (pgStatLocal.snapshot.per_backend_stats == NULL)
+ pgStatLocal.snapshot.per_backend_stats =
+ pgstat_per_backend_snapshot_create(pgStatLocal.snapshot.context, 64, NULL);
+
+ key.kind = kind;
+ key.procnum = procnum;
+
+ /* If already cached, return cached data */
+ entry = pgstat_per_backend_snapshot_lookup(pgStatLocal.snapshot.per_backend_stats, key);
+
+ if (entry)
+ return entry->data;
+
+ /* Insert new entry into the hash */
+ entry = pgstat_per_backend_snapshot_insert(pgStatLocal.snapshot.per_backend_stats,
+ key, &found);
+ Assert(!found);
+
+ if (data != NULL)
+ {
+ entry->data = MemoryContextAlloc(pgStatLocal.snapshot.context,
+ kind_info->per_backend_data_len);
+ memcpy(entry->data, data, kind_info->per_backend_data_len);
+ }
+ else
+ entry->data = NULL;
+
+ return entry->data;
+}
+
+static inline PgStat_PerBackendLocalState *
+pgstat_get_per_backend_local_state(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info PG_USED_FOR_ASSERTS_ONLY = pgstat_get_kind_info(kind);
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->fixed_amount);
+
+ if (!pgstat_is_kind_builtin(kind))
+ elog(ERROR, "invalid statistics kind: %u", kind);
+
+ return &per_backend_states[kind];
+}
+
+/*
+ * Create and cache this process's entry for one per-backend statistics kind.
+ */
+static PgStatShared_PerBackendEntry *
+pgstat_create_my_per_backend_entry(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ dshash_table *hash = pgstat_per_backend_attach(kind);
+ PgStatShared_PerBackendEntry *entry;
+ bool found;
+
+ Assert(state->my_entry == NULL);
+
+ if (hash == NULL)
+ return NULL;
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ entry = dshash_find_or_insert(hash, &MyProcNumber, &found);
+
+ /*
+ * A forced flush during early backend startup may already have created
+ * the entry. Preserve any statistics it contains.
+ */
+ if (!found)
+ {
+ entry->backend_type = MyBackendType;
+ LWLockInitialize(&entry->lock, LWTRANCHE_PGSTATS_DATA);
+ memset((char *) entry + kind_info->per_backend_data_off, 0,
+ kind_info->per_backend_data_len);
+ }
+
+ state->my_entry = entry;
+ dshash_release_lock(hash, entry);
+
+ return entry;
+}
+
+/*
+ * Create entries for all the kinds that use per-backend dshashes.
+ *
+ * Allocations and dshash resizes are deliberately done during backend
+ * initialization so routine stats flushes only need to conditionally acquire
+ * the cached entry's content lock.
+ */
+void
+pgstat_create_my_per_backend_entries(void)
+{
+ for (PgStat_Kind kind = PGSTAT_KIND_BUILTIN_MIN;
+ kind <= PGSTAT_KIND_BUILTIN_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (kind_info == NULL || kind_info->per_backend_data_len == 0)
+ continue;
+
+ (void) pgstat_create_my_per_backend_entry(kind);
+ }
+}
+
+/*
+ * Attach to the per-backend dshash for the given kind.
+ * Returns NULL if the hash is not available (e.g. during bootstrap or
+ * if this kind doesn't have per-backend tracking).
+ */
+dshash_table *
+pgstat_per_backend_attach(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table_handle *handle_ptr;
+ MemoryContext oldcontext;
+ dshash_parameters params;
+
+ if (state->hash != NULL)
+ return state->hash;
+
+ if (!kind_info || !kind_info->per_backend_data_len)
+ return NULL;
+
+ /* Get the shared struct for this kind */
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ handle_ptr = (dshash_table_handle *) (shared_struct + kind_info->per_backend_hash_handle_off);
+
+ if (*handle_ptr == DSHASH_HANDLE_INVALID)
+ return NULL;
+
+ /*
+ * Build dshash parameters from kind info. All per-backend hashes use
+ * ProcNumber keys and dshash_memcmp/dshash_memhash.
+ */
+ params.key_size = sizeof(ProcNumber);
+ params.entry_size = kind_info->per_backend_data_off + kind_info->per_backend_data_len;
+ params.compare_function = dshash_memcmp;
+ params.hash_function = dshash_memhash;
+ params.copy_function = dshash_memcpy;
+ params.tranche_id = LWTRANCHE_PGSTATS_HASH;
+
+ /* Attach in TopMemoryContext */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+ state->hash = dshash_attach(pgStatLocal.dsa, ¶ms, *handle_ptr, NULL);
+ MemoryContextSwitchTo(oldcontext);
+
+ return state->hash;
+}
+
+/*
+ * Lock this process's cached entry for a per-backend statistics kind.
+ *
+ * A missing entry is only created in the blocking path. The routine nowait
+ * path must not perform dshash lookups, allocations, or DSA address
+ * resolution.
+ */
+void *
+pgstat_lock_my_per_backend_entry(PgStat_Kind kind, bool nowait)
+{
+ const PgStat_KindInfo *kind_info PG_USED_FOR_ASSERTS_ONLY = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ PgStatShared_PerBackendEntry *entry = state->my_entry;
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ if (entry == NULL)
+ {
+ if (nowait)
+ return NULL;
+
+ entry = pgstat_create_my_per_backend_entry(kind);
+ if (entry == NULL)
+ return NULL;
+ }
+
+ if (nowait)
+ {
+ if (!LWLockConditionalAcquire(&entry->lock, LW_EXCLUSIVE))
+ return NULL;
+ }
+ else
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ return entry;
+}
+
+/*
+ * Accumulate all live per-backend entries into the kind's data in
+ * pgStatLocal.snapshot, optionally caching each entry for SNAPSHOT mode.
+ */
+void
+pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash, void *snap)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ dshash_seq_status hstat;
+ PgStatShared_PerBackendEntry *entry;
+
+ dshash_seq_init(&hstat, hash, false);
+
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->lock, LW_SHARED);
+
+ /* Kind-specific accumulation into global snapshot */
+ kind_info->per_backend_acc_cb(snap, entry);
+
+ /*
+ * In SNAPSHOT mode, cache each per-backend entry so that
+ * pgstat_fetch_per_backend() can return a consistent point-in-time
+ * view without re-reading from shared memory.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ {
+ pgstat_cache_per_backend_entry(kind, entry->key,
+ (char *) entry + kind_info->per_backend_data_off);
+ }
+
+ LWLockRelease(&entry->lock);
+ }
+
+ dshash_seq_term(&hstat);
+}
+
+/*
+ * Fetch per-backend stats for the given kind and ProcNumber.
+ * Returns NULL if no entry exists. In NONE mode, returns a copy allocated in
+ * the current memory context. In CACHE and SNAPSHOT modes, returns a pointer
+ * owned by the statistics snapshot cache.
+ */
+void *
+pgstat_fetch_per_backend(PgStat_Kind kind, ProcNumber procnum)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ dshash_table *hash;
+ PgStatShared_PerBackendEntry *entry;
+ void *stats_data;
+ PgStat_PerBackendSnapshotEntry *snapshot_entry;
+
+ pgstat_maybe_clear_snapshot();
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return NULL;
+
+ /* In NONE mode, read directly and don't cache */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
+ {
+ entry = dshash_find(hash, &procnum, false);
+ if (entry == NULL)
+ return NULL;
+
+ LWLockAcquire(&entry->lock, LW_SHARED);
+ stats_data = palloc(kind_info->per_backend_data_len);
+ memcpy(stats_data, (char *) entry + kind_info->per_backend_data_off,
+ kind_info->per_backend_data_len);
+ LWLockRelease(&entry->lock);
+ dshash_release_lock(hash, entry);
+
+ return stats_data;
+ }
+
+ Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE ||
+ pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT);
+
+ /*
+ * Building a full snapshot pre-caches all existing per-backend entries.
+ * CACHE mode only needs the requested entry, so it must not build the
+ * aggregate fixed-kind snapshot and scan every live backend.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ pgstat_snapshot_fixed(kind);
+
+ snapshot_entry = pgstat_lookup_per_backend_entry(kind, procnum);
+
+ if (snapshot_entry != NULL)
+ return snapshot_entry->data;
+
+ /*
+ * Once a full snapshot has been built, a cache miss means the entry did
+ * not exist at the snapshot point. Do not admit an entry created later.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ return NULL;
+
+ /* CACHE miss: copy the live entry directly into the snapshot context. */
+ entry = dshash_find(hash, &procnum, false);
+
+ if (entry == NULL)
+ return pgstat_cache_per_backend_entry(kind, procnum, NULL);
+
+ LWLockAcquire(&entry->lock, LW_SHARED);
+
+ stats_data = pgstat_cache_per_backend_entry(kind, procnum,
+ (char *) entry + kind_info->per_backend_data_off);
+
+ LWLockRelease(&entry->lock);
+ dshash_release_lock(hash, entry);
+
+ return stats_data;
+}
+
+/*
+ * Accumulate this process's per-backend stats into the global stats, then
+ * remove the entry from the dshash.
+ * Acquire the kind lock before the dshash partition lock. Snapshots and resets
+ * hold the kind lock while accessing both the global stats and live entries,
+ * so an entry cannot move between them during either operation.
+ *
+ * NB: The entry may belong to an earlier process that used the same ProcNumber.
+ * It must still be accumulated before removal.
+ */
+void
+pgstat_acc_my_per_backend(PgStat_Kind kind, LWLock *lock)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table *hash;
+ void *dst;
+ PgStatShared_PerBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return;
+
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ dst = shared_struct + kind_info->shared_data_off;
+
+ LWLockAcquire(lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &MyProcNumber, true);
+
+ if (entry == NULL)
+ {
+ state->my_entry = NULL;
+ LWLockRelease(lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ if (state->my_entry == entry)
+ state->my_entry = NULL;
+
+ /* Kind-specific accumulation */
+ kind_info->per_backend_acc_cb(dst, entry);
+
+ LWLockRelease(&entry->lock);
+
+ /* Remove the entry */
+ dshash_delete_entry(hash, entry);
+
+ LWLockRelease(lock);
+}
+
+/*
+ * Accumulate all per-backend entries into global stats and delete them.
+ * Called at clean server shutdown before writing the stats file. Acquire the
+ * kind's global lock before starting the dshash scan.
+ */
+void
+pgstat_acc_all_per_backend(PgStat_Kind kind, LWLock *lock)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table *hash;
+ void *dst;
+ dshash_seq_status hstat;
+ PgStatShared_PerBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return;
+
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ dst = shared_struct + kind_info->shared_data_off;
+
+ LWLockAcquire(lock, LW_EXCLUSIVE);
+
+ dshash_seq_init(&hstat, hash, true);
+
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ if (state->my_entry == entry)
+ state->my_entry = NULL;
+
+ /* Kind-specific accumulation */
+ kind_info->per_backend_acc_cb(dst, entry);
+
+ LWLockRelease(&entry->lock);
+ dshash_delete_current(&hstat);
+ }
+
+ dshash_seq_term(&hstat);
+
+ LWLockRelease(lock);
+}
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index d8ac9d6f5e8..a7e67f706c4 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -168,6 +168,47 @@ StatsShmemRequest(void *arg)
);
}
+/*
+ * Create a dshash for each built-in kind that stores per-backend statistics.
+ * Derive the entry size and handle location from the kind metadata, just as
+ * attachment does.
+ */
+static void
+pgstat_create_per_backend_hashes(dsa_area *dsa, PgStat_ShmemControl *ctl)
+{
+ dshash_parameters params = {
+ sizeof(ProcNumber),
+ 0,
+ dshash_memcmp,
+ dshash_memhash,
+ dshash_memcpy,
+ LWTRANCHE_PGSTATS_HASH
+ };
+
+ for (PgStat_Kind kind = PGSTAT_KIND_BUILTIN_MIN;
+ kind <= PGSTAT_KIND_BUILTIN_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ char *shared_struct;
+ dshash_table_handle *handle_ptr;
+ dshash_table *dsh;
+
+ if (kind_info == NULL || kind_info->per_backend_data_len == 0)
+ continue;
+
+ shared_struct = (char *) ctl + kind_info->shared_ctl_off;
+ handle_ptr = (dshash_table_handle *)
+ (shared_struct + kind_info->per_backend_hash_handle_off);
+
+ params.entry_size = kind_info->per_backend_data_off +
+ kind_info->per_backend_data_len;
+
+ dsh = dshash_create(dsa, ¶ms, NULL);
+ *handle_ptr = dshash_get_hash_table_handle(dsh);
+ dshash_detach(dsh);
+ }
+}
+
/*
* Initialize cumulative statistics system during startup
*/
@@ -210,6 +251,11 @@ StatsShmemInit(void *arg)
/* lift limit set above */
dsa_set_size_limit(dsa, -1);
+ /*
+ * Create per-backend hashes while the local DSA reference is available.
+ */
+ pgstat_create_per_backend_hashes(dsa, ctl);
+
/*
* Postmaster will never access these again, thus free the local
* dsa/dshash references.
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 1ef98620e1c..88d024cb4f3 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -380,6 +380,26 @@ typedef struct PgStat_KindInfo
*/
void (*snapshot_cb) (void);
+ /*
+ * Per-backend dshash support for built-in fixed-numbered statistics kinds
+ * that also maintain per-backend entries in a dedicated dshash. If
+ * per_backend_data_len is non-zero, the generic infrastructure handles
+ * attach, fetch, accumulate, and snapshot pre-caching automatically.
+ *
+ * Each entry starts with PgStatShared_PerBackendEntry, followed by the
+ * kind-specific statistics payload.
+ */
+ uint32 per_backend_data_off; /* offset of stats data in entry */
+ uint32 per_backend_data_len; /* size of stats data in entry */
+ /* offset of dshash_table_handle in shared struct */
+ uint32 per_backend_hash_handle_off;
+
+ /*
+ * Callback to accumulate one per-backend entry into a destination of the
+ * kind's statistics type. Called with the entry's content lock held.
+ */
+ void (*per_backend_acc_cb) (void *dst, void *entry);
+
/* name of the kind of stats */
const char *const name;
} PgStat_KindInfo;
@@ -479,6 +499,17 @@ typedef struct PgStatShared_SLRU
PgStat_SLRUStats stats[SLRU_NUM_ELEMENTS];
} PgStatShared_SLRU;
+/*
+ * Common header for entries in per-backend statistics dshashes. The
+ * ProcNumber key must be the first field for dshash.
+ */
+typedef struct PgStatShared_PerBackendEntry
+{
+ ProcNumber key;
+ BackendType backend_type;
+ LWLock lock;
+} PgStatShared_PerBackendEntry;
+
typedef struct PgStatShared_Wal
{
/* lock protects ->stats */
@@ -623,6 +654,9 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ /* Per-backend snapshot hash */
+ struct pgstat_per_backend_snapshot_hash *per_backend_stats;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -696,6 +730,17 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid,
bool *may_free);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void pgstat_maybe_clear_snapshot(void);
+
+/* Generic per-backend helpers */
+extern void pgstat_create_my_per_backend_entries(void);
+extern dshash_table *pgstat_per_backend_attach(PgStat_Kind kind);
+extern void *pgstat_lock_my_per_backend_entry(PgStat_Kind kind, bool nowait);
+extern void pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash,
+ void *snap);
+extern void *pgstat_fetch_per_backend(PgStat_Kind kind, ProcNumber procnum);
+extern void pgstat_acc_my_per_backend(PgStat_Kind kind, LWLock *lock);
+extern void pgstat_acc_all_per_backend(PgStat_Kind kind, LWLock *lock);
/*
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ff059c2a31e..7f16e37ea64 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2328,6 +2328,7 @@ PgStatShared_HashEntry
PgStatShared_Index
PgStatShared_IO
PgStatShared_Lock
+PgStatShared_PerBackendEntry
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2352,6 +2353,9 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_Lock
PgStat_LockEntry
+PgStat_PerBackendLocalState
+PgStat_PerBackendSnapshotEntry
+PgStat_PerBackendSnapshotKey
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
--
2.34.1
[text/x-diff] v2-0003-pgstat-move-WAL-statistics-to-new-per-backend-inf.patch (20.8K, ../../an3lxtz4TaYXXg%2F8@bdtpg/4-v2-0003-pgstat-move-WAL-statistics-to-new-per-backend-inf.patch)
download | inline diff:
From 34a65ad1b89a465e2004f7a7fffdb7955dbb6323 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 29 Jul 2026 14:13:28 +0000
Subject: [PATCH v2 3/6] pgstat: move WAL statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's WAL, Lock, and IO statistics
together in one variable-numbered entry keyed by ProcNumber. Move WAL
statistics into a dedicated ProcNumber keyed dshash associated with the
fixed WAL statistics kind. The global WAL stats hold data for backends that
have exited, while the dshash holds statistics for live backends.
Build pg_stat_wal snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove WAL counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_wal(). This also makes WAL statistics available for auxiliary
and shared memory-only workers. Update the documentation to describe the new
behavior.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
doc/src/sgml/monitoring.sgml | 7 +-
src/backend/utils/activity/pgstat.c | 18 ++
src/backend/utils/activity/pgstat_backend.c | 75 --------
src/backend/utils/activity/pgstat_wal.c | 179 +++++++++++++++++---
src/backend/utils/adt/pgstatfuncs.c | 37 +++-
src/include/pgstat.h | 3 +-
src/include/utils/pgstat_internal.h | 22 ++-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 232 insertions(+), 110 deletions(-)
3.6% doc/src/sgml/
78.0% src/backend/utils/activity/
10.2% src/backend/utils/adt/
6.0% src/include/utils/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 32cb6fdbd76..b6e5014588e 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5853,10 +5853,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns WAL statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_wal</structname> view.
- </para>
- <para>
- The function does not return WAL statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher.
</para></entry>
</row>
@@ -6014,7 +6010,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>wal</literal>: Reset all the counters shown in the
- <structname>pg_stat_wal</structname> view.
+ <structname>pg_stat_wal</structname> view, as well as per-backend
+ WAL statistics returned by <function>pg_stat_get_backend_wal</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 8e580fa7b89..446e687c0cc 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -516,6 +516,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_wal_init_shmem_cb,
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_WalBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_WalStats),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_Wal, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_wal_per_backend_acc_cb,
},
};
@@ -627,6 +632,9 @@ pgstat_before_server_shutdown(int code, Datum arg)
*/
if (code == 0)
{
+ /* Transfer all live per-backend stats before writing the stats file. */
+ pgstat_wal_acc_all_backends();
+
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
}
@@ -670,6 +678,9 @@ pgstat_shutdown_hook(int code, Datum arg)
if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
pgstat_request_entry_refs_gc();
+ /* Accumulate per-backend WAL stats into the global stats */
+ pgstat_wal_acc_backend_cb();
+
pgstat_detach_shmem();
#ifdef USE_ASSERT_CHECKING
@@ -690,6 +701,13 @@ pgstat_initialize(void)
pgstat_attach_shmem();
+ /*
+ * NB: need to accept that there might be stats from an older backend that
+ * used the same proc number. Accumulate them into the global stats before
+ * we start using the entry.
+ */
+ pgstat_wal_acc_backend_cb();
+
/*
* Create and cache per-backend statistics entries here. This also covers
* processes that never call InitPostgres(), such as shared-memory-only
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index b736b2ccc6f..76b970a8c41 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -41,14 +41,6 @@ static PgStat_BackendPending PendingBackendStats;
static bool backend_has_iostats = false;
static bool backend_has_lockstats = false;
-/*
- * WAL usage counters saved from pgWalUsage at the previous call to
- * pgstat_flush_backend(). This is used to calculate how much WAL usage
- * happens between pgstat_flush_backend() calls, by subtracting the
- * previous counters from the current ones.
- */
-static WalUsage prevBackendWalUsage;
-
/*
* Utility routines to report I/O stats for backends, kept here to avoid
* exposing PendingBackendStats to the outside world.
@@ -244,58 +236,6 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
backend_has_iostats = false;
}
-/*
- * To determine whether WAL usage happened.
- */
-static inline bool
-pgstat_backend_wal_have_pending(void)
-{
- return (pgWalUsage.wal_records != prevBackendWalUsage.wal_records);
-}
-
-/*
- * Flush out locally pending backend WAL statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_WalCounters *bktype_shstats;
- WalUsage wal_usage_diff = {0};
-
- /*
- * This function can be called even if nothing at all has happened for WAL
- * statistics. In this case, avoid unnecessarily modifying the stats
- * entry.
- */
- if (!pgstat_backend_wal_have_pending())
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.wal_counters;
-
- /*
- * Calculate how much WAL usage counters were increased by subtracting the
- * previous counters from the current ones.
- */
- WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevBackendWalUsage);
-
-#define WALSTAT_ACC(fld, var_to_add) \
- (bktype_shstats->fld += var_to_add.fld)
- WALSTAT_ACC(wal_buffers_full, wal_usage_diff);
- WALSTAT_ACC(wal_records, wal_usage_diff);
- WALSTAT_ACC(wal_fpi, wal_usage_diff);
- WALSTAT_ACC(wal_bytes, wal_usage_diff);
- WALSTAT_ACC(wal_fpi_bytes, wal_usage_diff);
-#undef WALSTAT_ACC
-
- /*
- * Save the current counters for the subsequent calculation of WAL usage.
- */
- prevBackendWalUsage = pgWalUsage;
-}
-
/*
* Flush out locally pending backend lock statistics. Locking is managed
* by the caller.
@@ -345,11 +285,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
has_pending_data = true;
- /* Some WAL data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_WAL) &&
- pgstat_backend_wal_have_pending())
- has_pending_data = true;
-
/* Some lock data pending? */
if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats)
has_pending_data = true;
@@ -366,9 +301,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_IO)
pgstat_flush_backend_entry_io(entry_ref);
- if (flags & PGSTAT_BACKEND_FLUSH_WAL)
- pgstat_flush_backend_entry_wal(entry_ref);
-
if (flags & PGSTAT_BACKEND_FLUSH_LOCK)
pgstat_flush_backend_entry_lock(entry_ref);
@@ -411,13 +343,6 @@ pgstat_create_backend(ProcNumber procnum)
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
backend_has_iostats = false;
backend_has_lockstats = false;
-
- /*
- * Initialize prevBackendWalUsage with pgWalUsage so that
- * pgstat_backend_flush_cb() can calculate how much pgWalUsage counters
- * are increased by subtracting prevBackendWalUsage from pgWalUsage.
- */
- prevBackendWalUsage = pgWalUsage;
}
/*
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..ece4d91ed70 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -8,6 +8,11 @@
* storage implementation and the details about individual types of
* statistics.
*
+ * WAL statistics use a per-backend dshash to avoid double-counting. Each
+ * backend flushes WAL usage to its own entry in the dshash (keyed by
+ * ProcNumber). The global pg_stat_wal view aggregates the global stats
+ * (which holds stats from exited backends) plus all live per-backend entries.
+ *
* Copyright (c) 2001-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -52,7 +57,6 @@ pgstat_report_wal(bool force)
/* flush wal stats */
(void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
/* flush IO stats */
pgstat_flush_io(nowait);
@@ -84,13 +88,15 @@ pgstat_wal_have_pending(void)
* Calculate how much WAL usage counters have increased by subtracting the
* previous counters from the current ones.
*
+ * Flush WAL usage counters to the per-backend dshash entry.
+ *
* If nowait is true, this function returns true if the lock could not be
* acquired. Otherwise return false.
*/
bool
pgstat_wal_flush_cb(bool nowait)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_WalBackendEntry *entry;
WalUsage wal_usage_diff = {0};
Assert(IsUnderPostmaster || !IsPostmasterEnvironment);
@@ -105,19 +111,18 @@ pgstat_wal_flush_cb(bool nowait)
return false;
/*
- * We don't update the WAL usage portion of the local WalStats elsewhere.
* Calculate how much WAL usage counters were increased by subtracting the
* previous counters from the current ones.
*/
WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevWalUsage);
- if (!nowait)
- LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
- return true;
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_WAL, nowait);
+
+ if (entry == NULL)
+ return nowait;
#define WALSTAT_ACC(fld, var_to_add) \
- (stats_shmem->stats.wal_counters.fld += var_to_add.fld)
+ (entry->stats.wal_counters.fld += var_to_add.fld)
WALSTAT_ACC(wal_records, wal_usage_diff);
WALSTAT_ACC(wal_fpi, wal_usage_diff);
WALSTAT_ACC(wal_bytes, wal_usage_diff);
@@ -125,7 +130,7 @@ pgstat_wal_flush_cb(bool nowait)
WALSTAT_ACC(wal_buffers_full, wal_usage_diff);
#undef WALSTAT_ACC
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&entry->header.lock);
/*
* Save the current counters for the subsequent calculation of WAL usage.
@@ -157,21 +162,157 @@ pgstat_wal_init_shmem_cb(void *stats)
void
pgstat_wal_reset_all_cb(TimestampTz ts)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ dshash_seq_status hstat;
+ PgStatShared_WalBackendEntry *entry;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
+
+ /* Reset all per-backend entries */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.wal_counters, 0, sizeof(PgStat_WalCounters));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
+ }
- LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
- memset(&stats_shmem->stats, 0, sizeof(stats_shmem->stats));
- stats_shmem->stats.stat_reset_timestamp = ts;
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build WAL stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_wal_snapshot_cb(void)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ PgStat_WalStats *snap = &pgStatLocal.snapshot.wal;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_WalStats));
+
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_WAL, hash, snap);
- LWLockAcquire(&stats_shmem->lock, LW_SHARED);
- memcpy(&pgStatLocal.snapshot.wal, &stats_shmem->stats,
- sizeof(pgStatLocal.snapshot.wal));
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&shmem->lock);
+}
+
+/* Macro to accumulate WAL counters from src into dst */
+#define WAL_ACCUMULATE_COUNTERS(dst, src) \
+do { \
+ (dst).wal_records += (src).wal_records; \
+ (dst).wal_fpi += (src).wal_fpi; \
+ (dst).wal_bytes += (src).wal_bytes; \
+ (dst).wal_fpi_bytes += (src).wal_fpi_bytes; \
+ (dst).wal_buffers_full += (src).wal_buffers_full; \
+} while (0)
+
+/*
+ * Accumulate one per-backend WAL entry into a snapshot or the global stats.
+ */
+void
+pgstat_wal_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_WalStats *stats = dst;
+ PgStatShared_WalBackendEntry *e = (PgStatShared_WalBackendEntry *) entry;
+
+ WAL_ACCUMULATE_COUNTERS(stats->wal_counters, e->stats.wal_counters);
+}
+
+/*
+ * Accumulate a backend's WAL stats into the global stats, then
+ * remove the entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_wal_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_WAL, &pgStatLocal.shmem->wal.lock);
+}
+
+/*
+ * Returns per-backend WAL statistics for the given ProcNumber.
+ */
+PgStat_WalStats *
+pgstat_fetch_stat_backend_wal(ProcNumber procnum)
+{
+ return (PgStat_WalStats *) pgstat_fetch_per_backend(PGSTAT_KIND_WAL, procnum);
+}
+
+/*
+ * Reset a backend's WAL stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_wal_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ dshash_table *hash;
+ PgStatShared_WalBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ WAL_ACCUMULATE_COUNTERS(shmem->stats.wal_counters, entry->stats.wal_counters);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats.wal_counters, 0, sizeof(PgStat_WalCounters));
+ entry->stats.stat_reset_timestamp = ts;
+
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
+}
+
+/*
+ * Accumulate all per-backend WAL stats entries into the global stats and remove
+ * them. Called at clean server shutdown to ensure all flushed data is preserved
+ * in the stats file.
+ */
+void
+pgstat_wal_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_WAL, &pgStatLocal.shmem->wal.lock);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 946025f39ed..789f96c6458 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1769,19 +1769,33 @@ Datum
pg_stat_get_backend_wal(PG_FUNCTION_ARGS)
{
int pid;
- PgStat_Backend *backend_stats;
- PgStat_WalCounters bktype_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ PgStat_WalStats *wal_stats;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, NULL);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
+ PG_RETURN_NULL();
+
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
PG_RETURN_NULL();
- bktype_stats = backend_stats->wal_counters;
+ wal_stats = pgstat_fetch_stat_backend_wal(procnum);
- /* save tuples with data from this PgStat_WalCounters */
- return (pg_stat_wal_build_tuple(bktype_stats, backend_stats->stat_reset_timestamp));
+ if (!wal_stats)
+ PG_RETURN_NULL();
+
+ return (pg_stat_wal_build_tuple(wal_stats->wal_counters,
+ wal_stats->stat_reset_timestamp));
}
/*
@@ -2174,6 +2188,7 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
PGPROC *proc;
PgBackendStatus *beentry;
ProcNumber procNumber;
+ TimestampTz ts;
int backend_pid = PG_GETARG_INT32(0);
proc = BackendPidGetProc(backend_pid);
@@ -2195,6 +2210,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
PG_RETURN_VOID();
+ /*
+ * Accumulate the backend's WAL stats into the global stats, then zero the
+ * entry.
+ */
+ ts = GetCurrentTimestamp();
+ pgstat_wal_reset_backend_cb(procNumber, ts);
+
+ /* Reset IO and Lock stats still in PGSTAT_KIND_BACKEND */
pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
PG_RETURN_VOID();
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 9c29dbe2c44..6c59056e15d 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -536,7 +536,6 @@ typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
PgStat_BktypeIO io_stats;
- PgStat_WalCounters wal_counters;
PgStat_PendingLock lock_stats;
} PgStat_Backend;
@@ -862,6 +861,8 @@ extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_
extern void pgstat_report_wal(bool force);
extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
+extern PgStat_WalStats *pgstat_fetch_stat_backend_wal(ProcNumber procnum);
+extern void pgstat_wal_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
/*
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 88d024cb4f3..644fe6ea594 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -510,11 +510,25 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for WAL statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_WalBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_WalStats stats;
+} PgStatShared_WalBackendEntry;
+
typedef struct PgStatShared_Wal
{
- /* lock protects ->stats */
LWLock lock;
PgStat_WalStats stats;
+
+ /*
+ * Per-backend dshash, keyed by ProcNumber.
+ */
+ dshash_table_handle backend_hash_handle;
} PgStatShared_Wal;
@@ -757,9 +771,8 @@ extern void pgstat_archiver_snapshot_cb(void);
/* flags for pgstat_flush_backend() */
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL | PGSTAT_BACKEND_FLUSH_LOCK)
+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_LOCK)
extern bool pgstat_flush_backend(bool nowait, uint32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
@@ -907,6 +920,9 @@ extern bool pgstat_wal_flush_cb(bool nowait);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
+extern void pgstat_wal_acc_backend_cb(void);
+extern void pgstat_wal_acc_all_backends(void);
+extern void pgstat_wal_per_backend_acc_cb(void *dst, void *entry);
/*
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7f16e37ea64..5c8b116ed31 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2334,6 +2334,7 @@ PgStatShared_ReplSlot
PgStatShared_SLRU
PgStatShared_Subscription
PgStatShared_Wal
+PgStatShared_WalBackendEntry
PgStat_ArchiverStats
PgStat_Backend
PgStat_BackendPending
--
2.34.1
[text/x-diff] v2-0004-pgstat-move-Lock-statistics-to-new-per-backend-in.patch (21.3K, ../../an3lxtz4TaYXXg%2F8@bdtpg/5-v2-0004-pgstat-move-Lock-statistics-to-new-per-backend-in.patch)
download | inline diff:
From 4fdccfc5553631b28653a68ced3c83ade0bc58d0 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:25:10 +0000
Subject: [PATCH v2 4/6] pgstat: move Lock statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's Lock and IO statistics together in one
variable-numbered entry keyed by ProcNumber. Move Lock statistics into a dedicated
ProcNumber keyed dshash associated with the fixed Lock statistics kind. The global
Lock stats hold data for backends that have exited, while the dshash holds
statistics for live backends.
Build pg_stat_lock snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove Lock counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_lock(). This also makes Lock statistics available for
auxiliary and shared memory-only workers. Update the documentation to describe the
new behavior.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/utils/activity/pgstat.c | 10 +-
src/backend/utils/activity/pgstat_backend.c | 72 --------
src/backend/utils/activity/pgstat_lock.c | 183 +++++++++++++++++---
src/backend/utils/adt/pgstatfuncs.c | 34 +++-
src/include/pgstat.h | 12 +-
src/include/utils/pgstat_internal.h | 21 ++-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 218 insertions(+), 123 deletions(-)
3.6% doc/src/sgml/
74.4% src/backend/utils/activity/
9.8% src/backend/utils/adt/
6.9% src/include/utils/
4.7% src/include/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b6e5014588e..4a625ba5259 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5834,10 +5834,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns lock statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_lock</structname> view.
- </para>
- <para>
- The function does not return lock statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher.
</para></entry>
</row>
@@ -5992,7 +5988,9 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>lock</literal>: Reset all the counters shown in the
- <structname>pg_stat_lock</structname> view.
+ <structname>pg_stat_lock</structname> view, as well as per-backend
+ lock statistics returned by
+ <function>pg_stat_get_backend_lock</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 446e687c0cc..2f5b3710644 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -481,6 +481,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_lock_init_shmem_cb,
.reset_all_cb = pgstat_lock_reset_all_cb,
.snapshot_cb = pgstat_lock_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_LockBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_Lock),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_Lock, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_lock_per_backend_acc_cb,
},
[PGSTAT_KIND_SLRU] = {
@@ -634,6 +639,7 @@ pgstat_before_server_shutdown(int code, Datum arg)
{
/* Transfer all live per-backend stats before writing the stats file. */
pgstat_wal_acc_all_backends();
+ pgstat_lock_acc_all_backends();
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
@@ -678,8 +684,9 @@ pgstat_shutdown_hook(int code, Datum arg)
if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
pgstat_request_entry_refs_gc();
- /* Accumulate per-backend WAL stats into the global stats */
+ /* Accumulate per-backend stats into the global stats */
pgstat_wal_acc_backend_cb();
+ pgstat_lock_acc_backend_cb();
pgstat_detach_shmem();
@@ -707,6 +714,7 @@ pgstat_initialize(void)
* we start using the entry.
*/
pgstat_wal_acc_backend_cb();
+ pgstat_lock_acc_backend_cb();
/*
* Create and cache per-backend statistics entries here. This also covers
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 76b970a8c41..8dda2cd88c1 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -39,7 +39,6 @@
*/
static PgStat_BackendPending PendingBackendStats;
static bool backend_has_iostats = false;
-static bool backend_has_lockstats = false;
/*
* Utility routines to report I/O stats for backends, kept here to avoid
@@ -79,39 +78,6 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
pgstat_report_fixed = true;
}
-/*
- * Utility routines to report lock stats for backends, kept here to avoid
- * exposing PendingBackendStats to the outside world.
- */
-void
-pgstat_count_backend_lock_waits(uint8 locktag_type, PgStat_Counter usecs)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(locktag_type <= LOCKTAG_LAST_TYPE);
-
- PendingBackendStats.pending_lock.stats[locktag_type].waits++;
- PendingBackendStats.pending_lock.stats[locktag_type].wait_time += usecs;
-
- backend_has_lockstats = true;
- pgstat_report_fixed = true;
-}
-
-void
-pgstat_count_backend_lock_fastpath_exceeded(uint8 locktag_type)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(locktag_type <= LOCKTAG_LAST_TYPE);
-
- PendingBackendStats.pending_lock.stats[locktag_type].fastpath_exceeded++;
-
- backend_has_lockstats = true;
- pgstat_report_fixed = true;
-}
-
/*
* Returns statistics of a backend by proc number.
*/
@@ -236,36 +202,6 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
backend_has_iostats = false;
}
-/*
- * Flush out locally pending backend lock statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_lock(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_PendingLock *bktype_shstats;
-
- if (!backend_has_lockstats)
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.lock_stats;
-
- for (int i = 0; i <= LOCKTAG_LAST_TYPE; i++)
- {
-#define LOCKSTAT_ACC(fld) \
- (bktype_shstats->stats[i].fld += PendingBackendStats.pending_lock.stats[i].fld)
- LOCKSTAT_ACC(waits);
- LOCKSTAT_ACC(wait_time);
- LOCKSTAT_ACC(fastpath_exceeded);
-#undef LOCKSTAT_ACC
- }
-
- MemSet(&PendingBackendStats.pending_lock, 0, sizeof(PgStat_PendingLock));
- backend_has_lockstats = false;
-}
-
/*
* Flush out locally pending backend statistics
*
@@ -285,10 +221,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
has_pending_data = true;
- /* Some lock data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats)
- has_pending_data = true;
-
if (!has_pending_data)
return false;
@@ -301,9 +233,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_IO)
pgstat_flush_backend_entry_io(entry_ref);
- if (flags & PGSTAT_BACKEND_FLUSH_LOCK)
- pgstat_flush_backend_entry_lock(entry_ref);
-
pgstat_unlock_entry(entry_ref);
return false;
@@ -342,7 +271,6 @@ pgstat_create_backend(ProcNumber procnum)
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
backend_has_iostats = false;
- backend_has_lockstats = false;
}
/*
diff --git a/src/backend/utils/activity/pgstat_lock.c b/src/backend/utils/activity/pgstat_lock.c
index c20c7599683..5aa380a6243 100644
--- a/src/backend/utils/activity/pgstat_lock.c
+++ b/src/backend/utils/activity/pgstat_lock.c
@@ -8,6 +8,12 @@
* access / storage implementation and the details about individual types
* of statistics.
*
+ * Lock statistics use a per-backend dshash to avoid double-counting. Each
+ * backend flushes lock stats (waits, wait_time, fastpath_exceeded per lock
+ * tag type) to its own entry in the dshash. The global pg_stat_lock view
+ * aggregates the global stats (which holds stats from exited backends) plus
+ * all live per-backend entries.
+ *
* Copyright (c) 2021-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -40,7 +46,7 @@ pgstat_lock_flush(bool nowait)
}
/*
- * Flush out locally pending lock statistics
+ * Flush out locally pending lock statistics to the per-backend dshash entry.
*
* If no stats have been recorded, this function returns false.
*
@@ -50,31 +56,27 @@ pgstat_lock_flush(bool nowait)
bool
pgstat_lock_flush_cb(bool nowait)
{
- LWLock *lckstat_lock;
- PgStatShared_Lock *shstats;
+ PgStatShared_LockBackendEntry *entry;
if (!have_lockstats)
return false;
- shstats = &pgStatLocal.shmem->lock;
- lckstat_lock = &shstats->lock;
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_LOCK, nowait);
- if (!nowait)
- LWLockAcquire(lckstat_lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(lckstat_lock, LW_EXCLUSIVE))
- return true;
+ if (entry == NULL)
+ return nowait;
for (int i = 0; i <= LOCKTAG_LAST_TYPE; i++)
{
#define LOCKSTAT_ACC(fld) \
- (shstats->stats.stats[i].fld += PendingLockStats.stats[i].fld)
+ (entry->stats.stats[i].fld += PendingLockStats.stats[i].fld)
LOCKSTAT_ACC(waits);
LOCKSTAT_ACC(wait_time);
LOCKSTAT_ACC(fastpath_exceeded);
#undef LOCKSTAT_ACC
}
- LWLockRelease(lckstat_lock);
+ LWLockRelease(&entry->header.lock);
memset(&PendingLockStats, 0, sizeof(PendingLockStats));
have_lockstats = false;
@@ -93,28 +95,125 @@ pgstat_lock_init_shmem_cb(void *stats)
void
pgstat_lock_reset_all_cb(TimestampTz ts)
{
- LWLock *lckstat_lock = &pgStatLocal.shmem->lock.lock;
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ dshash_seq_status hstat;
+ PgStatShared_LockBackendEntry *entry;
+ dshash_table *hash;
- LWLockAcquire(lckstat_lock, LW_EXCLUSIVE);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
- pgStatLocal.shmem->lock.stats.stat_reset_timestamp = ts;
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
- memset(pgStatLocal.shmem->lock.stats.stats, 0,
- sizeof(pgStatLocal.shmem->lock.stats.stats));
+ /* Reset all per-backend entries, since they contribute to the global view */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.stats, 0, sizeof(entry->stats.stats));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
+ }
- LWLockRelease(lckstat_lock);
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build lock stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_lock_snapshot_cb(void)
{
- LWLock *lckstat_lock = &pgStatLocal.shmem->lock.lock;
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ PgStat_Lock *snap = &pgStatLocal.snapshot.lock;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
+
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_Lock));
+
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_LOCK, hash, snap);
+
+ LWLockRelease(&shmem->lock);
+}
+
+/*
+ * Accumulate one per-backend lock entry into a snapshot or the global stats.
+ */
+void
+pgstat_lock_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_Lock *stats = dst;
+ PgStatShared_LockBackendEntry *e = (PgStatShared_LockBackendEntry *) entry;
+
+ for (int j = 0; j <= LOCKTAG_LAST_TYPE; j++)
+ {
+ stats->stats[j].waits += e->stats.stats[j].waits;
+ stats->stats[j].wait_time += e->stats.stats[j].wait_time;
+ stats->stats[j].fastpath_exceeded += e->stats.stats[j].fastpath_exceeded;
+ }
+}
+
+/* Macro to accumulate lock counters from src into dst */
+#define LOCK_ACCUMULATE_COUNTERS(dst, src) \
+do { \
+ for (int _i = 0; _i <= LOCKTAG_LAST_TYPE; _i++) \
+ { \
+ (dst)[_i].waits += (src)[_i].waits; \
+ (dst)[_i].wait_time += (src)[_i].wait_time; \
+ (dst)[_i].fastpath_exceeded += (src)[_i].fastpath_exceeded; \
+ } \
+} while (0)
- LWLockAcquire(lckstat_lock, LW_SHARED);
+/*
+ * Accumulate a backend's lock stats into the global stats, then
+ * remove the entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_lock_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_LOCK, &pgStatLocal.shmem->lock.lock);
+}
- pgStatLocal.snapshot.lock = pgStatLocal.shmem->lock.stats;
+/*
+ * Accumulate all per-backend lock stats entries into the global stats and remove
+ * them. Called at clean server shutdown to ensure all flushed data is preserved
+ * in the stats file.
+ */
+void
+pgstat_lock_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_LOCK, &pgStatLocal.shmem->lock.lock);
+}
- LWLockRelease(lckstat_lock);
+/*
+ * Returns per-backend lock statistics for the given ProcNumber.
+ */
+PgStat_Lock *
+pgstat_fetch_stat_backend_lock(ProcNumber procnum)
+{
+ return (PgStat_Lock *) pgstat_fetch_per_backend(PGSTAT_KIND_LOCK, procnum);
}
/*
@@ -131,8 +230,6 @@ pgstat_count_lock_fastpath_exceeded(uint8 locktag_type)
PendingLockStats.stats[locktag_type].fastpath_exceeded++;
have_lockstats = true;
pgstat_report_fixed = true;
-
- pgstat_count_backend_lock_fastpath_exceeded(locktag_type);
}
/*
@@ -149,6 +246,44 @@ pgstat_count_lock_waits(uint8 locktag_type, PgStat_Counter usecs)
PendingLockStats.stats[locktag_type].wait_time += usecs;
have_lockstats = true;
pgstat_report_fixed = true;
+}
+
+/*
+ * Reset a backend's lock stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_lock_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ dshash_table *hash;
+ PgStatShared_LockBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ LOCK_ACCUMULATE_COUNTERS(shmem->stats.stats, entry->stats.stats);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats.stats, 0, sizeof(entry->stats.stats));
+ entry->stats.stat_reset_timestamp = ts;
- pgstat_count_backend_lock_waits(locktag_type, usecs);
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 789f96c6458..502b281198e 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1872,19 +1872,36 @@ pg_stat_get_backend_lock(PG_FUNCTION_ARGS)
{
int pid;
ReturnSetInfo *rsinfo;
- PgStat_Backend *backend_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ PgStat_Lock *lock_stats;
InitMaterializedSRF(fcinfo, 0);
rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, NULL);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
return (Datum) 0;
- pg_stat_lock_build_tuples(rsinfo, backend_stats->lock_stats.stats,
- backend_stats->stat_reset_timestamp);
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
+ return (Datum) 0;
+
+ lock_stats = pgstat_fetch_stat_backend_lock(procnum);
+
+ if (!lock_stats)
+ return (Datum) 0;
+
+ pg_stat_lock_build_tuples(rsinfo, lock_stats->stats,
+ lock_stats->stat_reset_timestamp);
return (Datum) 0;
}
@@ -2211,13 +2228,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
/*
- * Accumulate the backend's WAL stats into the global stats, then zero the
- * entry.
+ * Accumulate the backend's WAL and lock stats into the global stats, then
+ * zero the entries.
*/
ts = GetCurrentTimestamp();
pgstat_wal_reset_backend_cb(procNumber, ts);
+ pgstat_lock_reset_backend_cb(procNumber, ts);
- /* Reset IO and Lock stats still in PGSTAT_KIND_BACKEND */
+ /* Reset IO stats still in PGSTAT_KIND_BACKEND */
pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
PG_RETURN_VOID();
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 6c59056e15d..2c1f42b923e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -536,7 +536,6 @@ typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
PgStat_BktypeIO io_stats;
- PgStat_PendingLock lock_stats;
} PgStat_Backend;
/* ---------
@@ -549,12 +548,6 @@ typedef struct PgStat_BackendPending
* Backend statistics store the same amount of IO data as PGSTAT_KIND_IO.
*/
PgStat_PendingIO pending_io;
-
- /*
- * Backend statistics store the same amount of lock data as
- * PGSTAT_KIND_LOCK.
- */
- PgStat_PendingLock pending_lock;
} PgStat_BackendPending;
/*
@@ -607,9 +600,6 @@ extern void pgstat_count_backend_io_op(IOObject io_object,
IOOp io_op, uint32 cnt,
uint64 bytes);
-/* used by pgstat_lock.c for lock stats tracked in backends */
-extern void pgstat_count_backend_lock_waits(uint8 locktag_type, PgStat_Counter usecs);
-extern void pgstat_count_backend_lock_fastpath_exceeded(uint8 locktag_type);
extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber);
extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid,
@@ -666,6 +656,8 @@ extern void pgstat_count_lock_fastpath_exceeded(uint8 locktag_type);
extern void pgstat_count_lock_waits(uint8 locktag_type,
PgStat_Counter usecs);
extern PgStat_Lock *pgstat_fetch_stat_lock(void);
+extern PgStat_Lock *pgstat_fetch_stat_backend_lock(ProcNumber procnum);
+extern void pgstat_lock_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
/*
* Functions in pgstat_database.c
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 644fe6ea594..800ef6c4c27 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -487,9 +487,12 @@ typedef struct PgStatShared_IO
typedef struct PgStatShared_Lock
{
- /* lock protects ->stats */
+ /* lock protects ->stats (global stats) */
LWLock lock;
PgStat_Lock stats;
+
+ /* Per-backend dshash, keyed by ProcNumber */
+ dshash_table_handle backend_hash_handle;
} PgStatShared_Lock;
typedef struct PgStatShared_SLRU
@@ -510,6 +513,16 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for lock statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_LockBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_Lock stats;
+} PgStatShared_LockBackendEntry;
+
/*
* Per-backend entry for WAL statistics, stored in a dshash keyed by
* ProcNumber.
@@ -771,8 +784,7 @@ extern void pgstat_archiver_snapshot_cb(void);
/* flags for pgstat_flush_backend() */
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_LOCK)
+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO)
extern bool pgstat_flush_backend(bool nowait, uint32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
@@ -838,6 +850,9 @@ extern bool pgstat_lock_flush_cb(bool nowait);
extern void pgstat_lock_init_shmem_cb(void *stats);
extern void pgstat_lock_reset_all_cb(TimestampTz ts);
extern void pgstat_lock_snapshot_cb(void);
+extern void pgstat_lock_acc_backend_cb(void);
+extern void pgstat_lock_acc_all_backends(void);
+extern void pgstat_lock_per_backend_acc_cb(void *dst, void *entry);
/*
* Functions in pgstat_relation.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 5c8b116ed31..bf960d8647b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2328,6 +2328,7 @@ PgStatShared_HashEntry
PgStatShared_Index
PgStatShared_IO
PgStatShared_Lock
+PgStatShared_LockBackendEntry
PgStatShared_PerBackendEntry
PgStatShared_Relation
PgStatShared_ReplSlot
--
2.34.1
[text/x-diff] v2-0005-pgstat-move-IO-statistics-to-new-per-backend-infr.patch (33.4K, ../../an3lxtz4TaYXXg%2F8@bdtpg/6-v2-0005-pgstat-move-IO-statistics-to-new-per-backend-infr.patch)
download | inline diff:
From 38d72b38a1ddb141d784fe31f3d40e26ce981800 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:26:07 +0000
Subject: [PATCH v2 5/6] pgstat: move IO statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's IO statistics in one variable-numbered
entry keyed by ProcNumber. Move IO statistics into a dedicated ProcNumber keyed
dshash associated with the fixed IO statistics kind. The global IO stats hold data
for backends that have exited, while the dshash holds statistics for live backends.
Build pg_stat_io snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove IO counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_io(). This also makes IO statistics available for auxiliary
and shared memory-only workers. Update the documentation to describe the
resulting new behavior.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
doc/src/sgml/monitoring.sgml | 9 +-
src/backend/replication/walsender.c | 2 -
src/backend/utils/activity/pgstat.c | 9 +-
src/backend/utils/activity/pgstat_backend.c | 155 +------------
src/backend/utils/activity/pgstat_io.c | 225 ++++++++++++++-----
src/backend/utils/activity/pgstat_relation.c | 2 -
src/backend/utils/activity/pgstat_shmem.c | 9 +-
src/backend/utils/activity/pgstat_wal.c | 1 -
src/backend/utils/adt/pgstatfuncs.c | 47 ++--
src/include/pgstat.h | 34 +--
src/include/utils/pgstat_internal.h | 28 ++-
src/test/regress/expected/stats.out | 23 +-
src/test/regress/sql/stats.sql | 11 +-
src/tools/pgindent/typedefs.list | 3 +-
14 files changed, 268 insertions(+), 290 deletions(-)
69.3% src/backend/utils/activity/
8.5% src/backend/utils/adt/
4.9% src/include/utils/
5.5% src/include/
4.3% src/test/regress/expected/
3.2% src/test/regress/sql/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 4a625ba5259..cf0c26194cb 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5797,12 +5797,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns I/O statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_io</structname> view.
- </para>
- <para>
- The function does not return I/O statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher
- as they are already visible in the <structname>pg_stat_io</structname>
- view and there is only one of each.
</para></entry>
</row>
@@ -5982,7 +5976,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>io</literal>: Reset all the counters shown in the
- <structname>pg_stat_io</structname> view.
+ <structname>pg_stat_io</structname> view, as well as per-backend
+ I/O statistics returned by <function>pg_stat_get_backend_io</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c65dd324325..6d4c15cad4c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -2083,7 +2083,6 @@ WalSndWaitForWal(XLogRecPtr loc)
WALSENDER_STATS_FLUSH_INTERVAL))
{
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
last_flush = now;
}
@@ -3177,7 +3176,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
WALSENDER_STATS_FLUSH_INTERVAL))
{
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
last_flush = now;
}
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 2f5b3710644..281104096c3 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -395,7 +395,6 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.shared_data_off = offsetof(PgStatShared_Backend, stats),
.shared_data_len = sizeof(((PgStatShared_Backend *) 0)->stats),
- .flush_static_cb = pgstat_backend_flush_cb,
.reset_timestamp_cb = pgstat_backend_reset_timestamp_cb,
},
@@ -464,6 +463,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_io_init_shmem_cb,
.reset_all_cb = pgstat_io_reset_all_cb,
.snapshot_cb = pgstat_io_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_IOBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_BackendIO),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_IO, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_io_per_backend_acc_cb,
},
[PGSTAT_KIND_LOCK] = {
@@ -640,6 +644,7 @@ pgstat_before_server_shutdown(int code, Datum arg)
/* Transfer all live per-backend stats before writing the stats file. */
pgstat_wal_acc_all_backends();
pgstat_lock_acc_all_backends();
+ pgstat_io_acc_all_backends();
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
@@ -687,6 +692,7 @@ pgstat_shutdown_hook(int code, Datum arg)
/* Accumulate per-backend stats into the global stats */
pgstat_wal_acc_backend_cb();
pgstat_lock_acc_backend_cb();
+ pgstat_io_acc_backend_cb();
pgstat_detach_shmem();
@@ -715,6 +721,7 @@ pgstat_initialize(void)
*/
pgstat_wal_acc_backend_cb();
pgstat_lock_acc_backend_cb();
+ pgstat_io_acc_backend_cb();
/*
* Create and cache per-backend statistics entries here. This also covers
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 8dda2cd88c1..b5996d15d8f 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -11,9 +11,7 @@
* This statistics kind uses a proc number as object ID for the hash table
* of pgstats. Entries are created each time a process is spawned, and are
* dropped when the process exits. These are not written to the pgstats file
- * on disk. Pending statistics are managed without direct interactions with
- * PgStat_EntryRef->pending, relying on PendingBackendStats instead so as it
- * is possible to report data within critical sections.
+ * on disk.
*
* Copyright (c) 2001-2026, PostgreSQL Global Development Group
*
@@ -24,60 +22,10 @@
#include "postgres.h"
-#include "access/xlog.h"
-#include "executor/instrument.h"
-#include "storage/bufmgr.h"
#include "storage/proc.h"
#include "storage/procarray.h"
-#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
-/*
- * Backend statistics counts waiting to be flushed out. These counters may be
- * reported within critical sections so we use static memory in order to avoid
- * memory allocation.
- */
-static PgStat_BackendPending PendingBackendStats;
-static bool backend_has_iostats = false;
-
-/*
- * Utility routines to report I/O stats for backends, kept here to avoid
- * exposing PendingBackendStats to the outside world.
- */
-void
-pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
- IOOp io_op, instr_time io_time)
-{
- Assert(track_io_timing || track_wal_io_timing);
-
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
-
- INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
- io_time);
-
- backend_has_iostats = true;
- pgstat_report_fixed = true;
-}
-
-void
-pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
- IOOp io_op, uint32 cnt, uint64 bytes)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
-
- PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
- PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
-
- backend_has_iostats = true;
- pgstat_report_fixed = true;
-}
-
/*
* Returns statistics of a backend by proc number.
*/
@@ -151,104 +99,6 @@ pgstat_fetch_stat_backend_by_pid(int pid, BackendType *bktype)
return backend_stats;
}
-/*
- * Flush out locally pending backend IO statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_BktypeIO *bktype_shstats;
- PgStat_PendingIO pending_io;
-
- /*
- * This function can be called even if nothing at all has happened for IO
- * statistics. In this case, avoid unnecessarily modifying the stats
- * entry.
- */
- if (!backend_has_iostats)
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.io_stats;
- pending_io = PendingBackendStats.pending_io;
-
- for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
- {
- for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
- {
- for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
- {
- instr_time time;
-
- bktype_shstats->counts[io_object][io_context][io_op] +=
- pending_io.counts[io_object][io_context][io_op];
- bktype_shstats->bytes[io_object][io_context][io_op] +=
- pending_io.bytes[io_object][io_context][io_op];
- time = pending_io.pending_times[io_object][io_context][io_op];
-
- bktype_shstats->times[io_object][io_context][io_op] +=
- INSTR_TIME_GET_MICROSEC(time);
- }
- }
- }
-
- /*
- * Clear out the statistics buffer, so it can be re-used.
- */
- MemSet(&PendingBackendStats.pending_io, 0, sizeof(PgStat_PendingIO));
-
- backend_has_iostats = false;
-}
-
-/*
- * Flush out locally pending backend statistics
- *
- * "flags" parameter controls which statistics to flush. Returns true
- * if some statistics could not be flushed due to lock contention.
- */
-bool
-pgstat_flush_backend(bool nowait, uint32 flags)
-{
- PgStat_EntryRef *entry_ref;
- bool has_pending_data = false;
-
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return false;
-
- /* Some IO data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
- has_pending_data = true;
-
- if (!has_pending_data)
- return false;
-
- entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid,
- MyProcNumber, nowait);
- if (!entry_ref)
- return true;
-
- /* Flush requested statistics */
- if (flags & PGSTAT_BACKEND_FLUSH_IO)
- pgstat_flush_backend_entry_io(entry_ref);
-
- pgstat_unlock_entry(entry_ref);
-
- return false;
-}
-
-/*
- * Callback to flush out locally pending backend statistics.
- *
- * If some stats could not be flushed due to lock contention, return true.
- */
-bool
-pgstat_backend_flush_cb(bool nowait)
-{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
-}
-
/*
* Create backend statistics entry for proc number.
*/
@@ -268,9 +118,6 @@ pgstat_create_backend(ProcNumber procnum)
*/
memset(&shstatent->stats, 0, sizeof(shstatent->stats));
pgstat_unlock_entry(entry_ref);
-
- MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
- backend_has_iostats = false;
}
/*
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 8ec1aad5078..5e65bc5cb4b 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -7,6 +7,11 @@
* from pgstat.c to enforce the line between the statistics access / storage
* implementation and the details about individual types of statistics.
*
+ * IO statistics use a per-backend dshash to avoid double-counting. Each
+ * process flushes IO stats to its own entry in the dshash (keyed by
+ * ProcNumber). The global pg_stat_io view aggregates the global stats
+ * (which holds stats from exited processes) plus all live per-backend entries.
+ *
* Copyright (c) 2021-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -68,9 +73,6 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
PendingIOStats.counts[io_object][io_context][io_op] += cnt;
PendingIOStats.bytes[io_object][io_context][io_op] += bytes;
- /* Add the per-backend counts */
- pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
-
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -143,10 +145,6 @@ pgstat_count_io_op_time(IOObject io_object, IOContext io_context, IOOp io_op,
INSTR_TIME_ADD(PendingIOStats.pending_times[io_object][io_context][io_op],
io_time);
-
- /* Add the per-backend count */
- pgstat_count_backend_io_op_time(io_object, io_context, io_op,
- io_time);
}
pgstat_count_io_op(io_object, io_context, io_op, cnt, bytes);
@@ -170,7 +168,7 @@ pgstat_flush_io(bool nowait)
}
/*
- * Flush out locally pending IO statistics
+ * Flush out locally pending IO statistics to the per-backend dshash entry.
*
* If no stats have been recorded, this function returns false.
*
@@ -180,20 +178,18 @@ pgstat_flush_io(bool nowait)
bool
pgstat_io_flush_cb(bool nowait)
{
- LWLock *bktype_lock;
+ PgStatShared_IOBackendEntry *entry;
PgStat_BktypeIO *bktype_shstats;
if (!have_iostats)
return false;
- bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
- bktype_shstats =
- &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_IO, nowait);
- if (!nowait)
- LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
- return true;
+ if (entry == NULL)
+ return nowait;
+
+ bktype_shstats = &entry->stats.stats;
for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
{
@@ -217,12 +213,9 @@ pgstat_io_flush_cb(bool nowait)
}
}
- Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
-
- LWLockRelease(bktype_lock);
+ LWLockRelease(&entry->header.lock);
memset(&PendingIOStats, 0, sizeof(PendingIOStats));
-
have_iostats = false;
return false;
@@ -271,55 +264,70 @@ pgstat_io_init_shmem_cb(void *stats)
{
PgStatShared_IO *stat_shmem = (PgStatShared_IO *) stats;
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- LWLockInitialize(&stat_shmem->locks[i], LWTRANCHE_PGSTATS_DATA);
+ LWLockInitialize(&stat_shmem->lock, LWTRANCHE_PGSTATS_DATA);
}
void
pgstat_io_reset_all_cb(TimestampTz ts)
{
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- {
- LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
- PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ dshash_seq_status hstat;
+ PgStatShared_IOBackendEntry *entry;
+ dshash_table *hash;
- LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
- /*
- * Use the lock in the first BackendType's PgStat_BktypeIO to protect
- * the reset timestamp as well.
- */
- if (i == 0)
- pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
- memset(bktype_shstats, 0, sizeof(*bktype_shstats));
- LWLockRelease(bktype_lock);
+ /* Reset all per-backend entries */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.stats, 0, sizeof(PgStat_BktypeIO));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
}
+
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build IO stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_io_snapshot_cb(void)
{
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- {
- LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
- PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
- PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ PgStat_IO *snap = &pgStatLocal.snapshot.io;
+ dshash_table *hash;
- LWLockAcquire(bktype_lock, LW_SHARED);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
- /*
- * Use the lock in the first BackendType's PgStat_BktypeIO to protect
- * the reset timestamp as well.
- */
- if (i == 0)
- pgStatLocal.snapshot.io.stat_reset_timestamp =
- pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_IO));
- /* using struct assignment due to better type safety */
- *bktype_snap = *bktype_shstats;
- LWLockRelease(bktype_lock);
- }
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_IO, hash, snap);
+
+ LWLockRelease(&shmem->lock);
}
/*
@@ -581,3 +589,116 @@ pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
return true;
}
+
+/*
+ * Accumulate IO counters from src into dst.
+ */
+static inline void
+pgstat_io_accumulate_counters(PgStat_BktypeIO *dst, const PgStat_BktypeIO *src)
+{
+ for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ dst->counts[io_object][io_context][io_op] +=
+ src->counts[io_object][io_context][io_op];
+ dst->bytes[io_object][io_context][io_op] +=
+ src->bytes[io_object][io_context][io_op];
+ dst->times[io_object][io_context][io_op] +=
+ src->times[io_object][io_context][io_op];
+ }
+ }
+ }
+}
+
+/*
+ * Accumulate one per-backend IO entry into a snapshot or the global stats.
+ */
+void
+pgstat_io_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_IO *stats = dst;
+ PgStatShared_IOBackendEntry *e = (PgStatShared_IOBackendEntry *) entry;
+ BackendType bktype = e->header.backend_type;
+
+ if (bktype == B_INVALID)
+ return;
+
+ pgstat_io_accumulate_counters(&stats->stats[bktype], &e->stats.stats);
+}
+
+/*
+ * Accumulate a backend's IO stats into the global stats, then remove the
+ * entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_io_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_IO, &pgStatLocal.shmem->io.lock);
+}
+
+/*
+ * Accumulate all remaining per-backend IO stats entries into the global stats
+ * and remove them. Called at clean server shutdown to ensure all flushed data
+ * is preserved in the stats file.
+ */
+void
+pgstat_io_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_IO, &pgStatLocal.shmem->io.lock);
+}
+
+/*
+ * Returns per-backend IO statistics for the given ProcNumber.
+ */
+PgStat_BackendIO *
+pgstat_fetch_stat_backend_io(ProcNumber procnum)
+{
+ return (PgStat_BackendIO *) pgstat_fetch_per_backend(PGSTAT_KIND_IO, procnum);
+}
+
+/*
+ * Reset a backend's IO stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_io_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ dshash_table *hash;
+ PgStatShared_IOBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ pgstat_io_accumulate_counters(&shmem->stats.stats[entry->header.backend_type],
+ &entry->stats.stats);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats, 0, sizeof(entry->stats));
+ entry->stats.stat_reset_timestamp = ts;
+
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index beea9188773..e4eccaef595 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -316,7 +316,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* VACUUM command has processed all tables and committed.
*/
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -411,7 +410,6 @@ pgstat_report_analyze(Relation rel,
/* see pgstat_report_vacuum() */
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index a7e67f706c4..993024eaea5 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -118,12 +118,11 @@ pgstat_dsa_init_size(void)
/*
* The dshash header / initial buckets array needs to fit into "plain"
* shared memory, but it's beneficial to not need dsm segments
- * immediately. A size of 256kB seems works well and is not
- * disproportional compared to other constant sized shared memory
- * allocations. NB: To avoid DSMs further, the user can configure
- * min_dynamic_shared_memory.
+ * immediately. A size of 1MB works well and is not disproportional
+ * compared to other constant sized shared memory allocations. NB: To
+ * avoid DSMs further, the user can configure min_dynamic_shared_memory.
*/
- sz = 256 * 1024;
+ sz = 1024 * 1024;
Assert(dsa_minimum_size() <= sz);
return MAXALIGN(sz);
}
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index ece4d91ed70..99fd20a04c2 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -60,7 +60,6 @@ pgstat_report_wal(bool force)
/* flush IO stats */
pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 502b281198e..a054c7bda69 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1671,32 +1671,47 @@ Datum
pg_stat_get_backend_io(PG_FUNCTION_ARGS)
{
ReturnSetInfo *rsinfo;
- BackendType bktype;
int pid;
- PgStat_Backend *backend_stats;
- PgStat_BktypeIO *bktype_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ BackendType bktype;
+ PgStat_BackendIO *backend_io;
InitMaterializedSRF(fcinfo, 0);
rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, &bktype);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
+ return (Datum) 0;
+
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
return (Datum) 0;
- bktype_stats = &backend_stats->io_stats;
+ bktype = beentry->st_backendType;
+ backend_io = pgstat_fetch_stat_backend_io(procnum);
+
+ if (!backend_io)
+ return (Datum) 0;
/*
* In Assert builds, we can afford an extra loop through all of the
* counters (in pg_stat_io_build_tuples()), checking that only expected
* stats are non-zero, since it keeps the non-Assert code cleaner.
*/
- Assert(pgstat_bktype_io_stats_valid(bktype_stats, bktype));
+ Assert(pgstat_bktype_io_stats_valid(&backend_io->stats, bktype));
+
+ pg_stat_io_build_tuples(rsinfo, &backend_io->stats, bktype,
+ backend_io->stat_reset_timestamp);
- /* save tuples with data from this PgStat_BktypeIO */
- pg_stat_io_build_tuples(rsinfo, bktype_stats, bktype,
- backend_stats->stat_reset_timestamp);
return (Datum) 0;
}
@@ -2223,20 +2238,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
if (!beentry)
PG_RETURN_VOID();
- /* Check if the backend type tracks statistics */
- if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
- PG_RETURN_VOID();
-
/*
- * Accumulate the backend's WAL and lock stats into the global stats, then
- * zero the entries.
+ * Accumulate the backend's WAL, lock and IO stats into the global stats,
+ * then zero the entries.
*/
ts = GetCurrentTimestamp();
pgstat_wal_reset_backend_cb(procNumber, ts);
pgstat_lock_reset_backend_cb(procNumber, ts);
-
- /* Reset IO stats still in PGSTAT_KIND_BACKEND */
- pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
+ pgstat_io_reset_backend_cb(procNumber, ts);
PG_RETURN_VOID();
}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2c1f42b923e..cd6bfc9a33b 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -218,7 +218,7 @@ typedef struct PgStat_TableXactStatus
* ------------------------------------------------------------
*/
-#define PGSTAT_FILE_FORMAT_ID 0x01A5BCBD
+#define PGSTAT_FILE_FORMAT_ID 0x01A5BCBE
typedef struct PgStat_ArchiverStats
{
@@ -333,6 +333,12 @@ typedef struct PgStat_BktypeIO
PgStat_Counter times[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
} PgStat_BktypeIO;
+typedef struct PgStat_BackendIO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BktypeIO stats;
+} PgStat_BackendIO;
+
typedef struct PgStat_PendingIO
{
uint64 bytes[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
@@ -535,21 +541,8 @@ typedef struct PgStat_WalStats
typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
- PgStat_BktypeIO io_stats;
} PgStat_Backend;
-/* ---------
- * PgStat_BackendPending Non-flushed backend stats.
- * ---------
- */
-typedef struct PgStat_BackendPending
-{
- /*
- * Backend statistics store the same amount of IO data as PGSTAT_KIND_IO.
- */
- PgStat_PendingIO pending_io;
-} PgStat_BackendPending;
-
/*
* Functions in pgstat.c
*/
@@ -590,17 +583,6 @@ extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void);
* Functions in pgstat_backend.c
*/
-/* used by pgstat_io.c for I/O stats tracked in backends */
-extern void pgstat_count_backend_io_op_time(IOObject io_object,
- IOContext io_context,
- IOOp io_op,
- instr_time io_time);
-extern void pgstat_count_backend_io_op(IOObject io_object,
- IOContext io_context,
- IOOp io_op, uint32 cnt,
- uint64 bytes);
-
-
extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber);
extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid,
BackendType *bktype);
@@ -637,6 +619,8 @@ extern void pgstat_count_io_op_time(IOObject io_object, IOContext io_context,
uint32 cnt, uint64 bytes);
extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern PgStat_BackendIO *pgstat_fetch_stat_backend_io(ProcNumber procnum);
+extern void pgstat_io_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
extern const char *pgstat_get_io_context_name(IOContext io_context);
extern const char *pgstat_get_io_object_name(IOObject io_object);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 800ef6c4c27..95d52e1a26e 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -477,12 +477,13 @@ typedef struct PgStatShared_Checkpointer
/* Shared-memory ready PgStat_IO */
typedef struct PgStatShared_IO
{
+ LWLock lock;
+ PgStat_IO stats;
+
/*
- * locks[i] protects stats.stats[i]. locks[0] also protects
- * stats.stat_reset_timestamp.
+ * Per-backend dshash, keyed by ProcNumber.
*/
- LWLock locks[BACKEND_NUM_TYPES];
- PgStat_IO stats;
+ dshash_table_handle backend_hash_handle;
} PgStatShared_IO;
typedef struct PgStatShared_Lock
@@ -513,6 +514,16 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for IO statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_IOBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_BackendIO stats;
+} PgStatShared_IOBackendEntry;
+
/*
* Per-backend entry for lock statistics, stored in a dshash keyed by
* ProcNumber.
@@ -782,12 +793,6 @@ extern void pgstat_archiver_snapshot_cb(void);
* Functions in pgstat_backend.c
*/
-/* flags for pgstat_flush_backend() */
-#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO)
-
-extern bool pgstat_flush_backend(bool nowait, uint32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -841,6 +846,9 @@ extern bool pgstat_io_flush_cb(bool nowait);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
+extern void pgstat_io_acc_backend_cb(void);
+extern void pgstat_io_acc_all_backends(void);
+extern void pgstat_io_per_backend_acc_cb(void *dst, void *entry);
/*
* Functions in pgstat_lock.c
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 9db59e25210..4255c1234f4 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1871,13 +1871,20 @@ SELECT :io_stats_post_reset < :io_stats_pre_reset;
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset
FROM pg_stat_get_backend_io(pg_backend_pid()) \gset
--- pg_stat_reset_shared() did not reset backend IO stats
-SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset;
+-- pg_stat_reset_shared() also resets per-backend IO stats
+SELECT :my_io_stats_pre_reset > :my_io_stats_post_reset;
?column?
----------
t
(1 row)
+SELECT bool_and(stats_reset IS NOT NULL) AS backend_io_reset_timestamp_set
+ FROM pg_stat_get_backend_io(pg_backend_pid());
+ backend_io_reset_timestamp_set
+--------------------------------
+ t
+(1 row)
+
-- but pg_stat_reset_backend_stats() does
SELECT pg_stat_reset_backend_stats(pg_backend_pid());
pg_stat_reset_backend_stats
@@ -1904,11 +1911,13 @@ SELECT pg_stat_get_backend_io(0);
------------------------
(0 rows)
--- Auxiliary processes return no data.
-SELECT pg_stat_get_backend_io(:checkpointer_pid);
- pg_stat_get_backend_io
-------------------------
-(0 rows)
+-- Auxiliary processes now return data
+SELECT count(*) > 0 AS checkpointer_has_io_stats
+ FROM pg_stat_get_backend_io(:checkpointer_pid);
+ checkpointer_has_io_stats
+---------------------------
+ t
+(1 row)
-- test BRIN index doesn't block HOT update
CREATE TABLE brin_hot (
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 176bd008673..12cac8ebb0b 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -878,8 +878,10 @@ SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) +
SELECT :io_stats_post_reset < :io_stats_pre_reset;
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset
FROM pg_stat_get_backend_io(pg_backend_pid()) \gset
--- pg_stat_reset_shared() did not reset backend IO stats
-SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset;
+-- pg_stat_reset_shared() also resets per-backend IO stats
+SELECT :my_io_stats_pre_reset > :my_io_stats_post_reset;
+SELECT bool_and(stats_reset IS NOT NULL) AS backend_io_reset_timestamp_set
+ FROM pg_stat_get_backend_io(pg_backend_pid());
-- but pg_stat_reset_backend_stats() does
SELECT pg_stat_reset_backend_stats(pg_backend_pid());
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_backend_reset
@@ -889,8 +891,9 @@ SELECT :my_io_stats_pre_reset > :my_io_stats_post_backend_reset;
-- Check invalid input for pg_stat_get_backend_io()
SELECT pg_stat_get_backend_io(NULL);
SELECT pg_stat_get_backend_io(0);
--- Auxiliary processes return no data.
-SELECT pg_stat_get_backend_io(:checkpointer_pid);
+-- Auxiliary processes now return data
+SELECT count(*) > 0 AS checkpointer_has_io_stats
+ FROM pg_stat_get_backend_io(:checkpointer_pid);
-- test BRIN index doesn't block HOT update
CREATE TABLE brin_hot (
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bf960d8647b..48a1deb9bca 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2327,6 +2327,7 @@ PgStatShared_Function
PgStatShared_HashEntry
PgStatShared_Index
PgStatShared_IO
+PgStatShared_IOBackendEntry
PgStatShared_Lock
PgStatShared_LockBackendEntry
PgStatShared_PerBackendEntry
@@ -2338,7 +2339,7 @@ PgStatShared_Wal
PgStatShared_WalBackendEntry
PgStat_ArchiverStats
PgStat_Backend
-PgStat_BackendPending
+PgStat_BackendIO
PgStat_BackendSubEntry
PgStat_BgWriterStats
PgStat_BktypeIO
--
2.34.1
[text/x-diff] v2-0006-pgstat-remove-PGSTAT_KIND_BACKEND.patch (13.9K, ../../an3lxtz4TaYXXg%2F8@bdtpg/7-v2-0006-pgstat-remove-PGSTAT_KIND_BACKEND.patch)
download | inline diff:
From 5c68c49558f79b9783bf1423af6420680e229770 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bdrouvot@amazon.com>
Date: Thu, 13 Aug 2026 12:28:55 +0000
Subject: [PATCH v2 6/6] pgstat: remove PGSTAT_KIND_BACKEND
After WAL, Lock, and IO have moved, PGSTAT_KIND_BACKEND contains no data.
Remove that kind and pgstat_backend.c together with their build entries,
shared structures, callbacks, and call sites.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
src/backend/utils/activity/Makefile | 1 -
src/backend/utils/activity/backend_status.c | 4 -
src/backend/utils/activity/meson.build | 1 -
src/backend/utils/activity/pgstat.c | 20 ---
src/backend/utils/activity/pgstat_backend.c | 179 --------------------
src/include/pgstat.h | 19 ---
src/include/utils/pgstat_internal.h | 13 --
src/include/utils/pgstat_kind.h | 15 +-
src/test/regress/expected/stats.out | 17 +-
src/tools/pgindent/typedefs.list | 2 -
10 files changed, 15 insertions(+), 256 deletions(-)
72.8% src/backend/utils/activity/
9.8% src/include/utils/
6.4% src/include/
10.3% src/test/regress/expected/
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index 8924b10ed2e..618a04602c8 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -20,7 +20,6 @@ OBJS = \
backend_status.o \
pgstat.o \
pgstat_archiver.o \
- pgstat_backend.o \
pgstat_bgwriter.o \
pgstat_checkpointer.o \
pgstat_database.o \
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index d685fc5cd87..7088fc09651 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -458,10 +458,6 @@ pgstat_bestart_final(void)
PGSTAT_END_WRITE_ACTIVITY(beentry);
- /* Create the backend statistics entry */
- if (pgstat_tracks_backend_bktype(MyBackendType))
- pgstat_create_backend(MyProcNumber);
-
/* Update app name to current GUC setting */
if (application_name)
pgstat_report_appname(application_name);
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 6434c377e53..ea6495b95d3 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -5,7 +5,6 @@ backend_sources += files(
'backend_status.c',
'pgstat.c',
'pgstat_archiver.c',
- 'pgstat_backend.c',
'pgstat_bgwriter.c',
'pgstat_checkpointer.c',
'pgstat_database.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 281104096c3..aad5cef2a3f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -78,7 +78,6 @@
*
* Each statistics kind is handled in a dedicated file:
* - pgstat_archiver.c
- * - pgstat_backend.c
* - pgstat_bgwriter.c
* - pgstat_checkpointer.c
* - pgstat_database.c
@@ -383,21 +382,6 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_timestamp_cb = pgstat_subscription_reset_timestamp_cb,
},
- [PGSTAT_KIND_BACKEND] = {
- .name = "backend",
-
- .fixed_amount = false,
- .write_to_file = false,
-
- .accessed_across_databases = true,
-
- .shared_size = sizeof(PgStatShared_Backend),
- .shared_data_off = offsetof(PgStatShared_Backend, stats),
- .shared_data_len = sizeof(((PgStatShared_Backend *) 0)->stats),
-
- .reset_timestamp_cb = pgstat_backend_reset_timestamp_cb,
- },
-
/* stats for fixed-numbered (mostly 1) objects */
[PGSTAT_KIND_ARCHIVER] = {
@@ -685,10 +669,6 @@ pgstat_shutdown_hook(int code, Datum arg)
Assert(dlist_is_empty(&pgStatPending));
dlist_init(&pgStatPending);
- /* drop the backend stats entry */
- if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
- pgstat_request_entry_refs_gc();
-
/* Accumulate per-backend stats into the global stats */
pgstat_wal_acc_backend_cb();
pgstat_lock_acc_backend_cb();
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
deleted file mode 100644
index b5996d15d8f..00000000000
--- a/src/backend/utils/activity/pgstat_backend.c
+++ /dev/null
@@ -1,179 +0,0 @@
-/* -------------------------------------------------------------------------
- *
- * pgstat_backend.c
- * Implementation of backend statistics.
- *
- * This file contains the implementation of backend statistics. It is kept
- * separate from pgstat.c to enforce the line between the statistics access /
- * storage implementation and the details about individual types of
- * statistics.
- *
- * This statistics kind uses a proc number as object ID for the hash table
- * of pgstats. Entries are created each time a process is spawned, and are
- * dropped when the process exits. These are not written to the pgstats file
- * on disk.
- *
- * Copyright (c) 2001-2026, PostgreSQL Global Development Group
- *
- * IDENTIFICATION
- * src/backend/utils/activity/pgstat_backend.c
- * -------------------------------------------------------------------------
- */
-
-#include "postgres.h"
-
-#include "storage/proc.h"
-#include "storage/procarray.h"
-#include "utils/pgstat_internal.h"
-
-/*
- * Returns statistics of a backend by proc number.
- */
-PgStat_Backend *
-pgstat_fetch_stat_backend(ProcNumber procNumber)
-{
- PgStat_Backend *backend_entry;
-
- backend_entry = (PgStat_Backend *) pgstat_fetch_entry(PGSTAT_KIND_BACKEND,
- InvalidOid, procNumber,
- NULL);
-
- return backend_entry;
-}
-
-/*
- * Returns statistics of a backend by pid.
- *
- * This routine includes sanity checks to ensure that the backend exists and
- * is running. "bktype" can be optionally defined to return the BackendType
- * of the backend whose statistics are returned.
- */
-PgStat_Backend *
-pgstat_fetch_stat_backend_by_pid(int pid, BackendType *bktype)
-{
- PGPROC *proc;
- PgBackendStatus *beentry;
- ProcNumber procNumber;
- PgStat_Backend *backend_stats;
-
- proc = BackendPidGetProc(pid);
- if (bktype)
- *bktype = B_INVALID;
-
- /* this could be an auxiliary process */
- if (!proc)
- proc = AuxiliaryPidGetProc(pid);
-
- if (!proc)
- return NULL;
-
- procNumber = GetNumberFromPGProc(proc);
-
- beentry = pgstat_get_beentry_by_proc_number(procNumber);
- if (!beentry)
- return NULL;
-
- /* check if the backend type tracks statistics */
- if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
- return NULL;
-
- /* if PID does not match, leave */
- if (beentry->st_procpid != pid)
- return NULL;
-
- if (bktype)
- *bktype = beentry->st_backendType;
-
- /*
- * Retrieve the entry. Note that "beentry" may be freed depending on the
- * value of stats_fetch_consistency, so do not access it from this point.
- */
- backend_stats = pgstat_fetch_stat_backend(procNumber);
- if (!backend_stats)
- {
- if (bktype)
- *bktype = B_INVALID;
- return NULL;
- }
-
- return backend_stats;
-}
-
-/*
- * Create backend statistics entry for proc number.
- */
-void
-pgstat_create_backend(ProcNumber procnum)
-{
- PgStat_EntryRef *entry_ref;
- PgStatShared_Backend *shstatent;
-
- entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid,
- procnum, false);
- shstatent = (PgStatShared_Backend *) entry_ref->shared_stats;
-
- /*
- * NB: need to accept that there might be stats from an older backend,
- * e.g. if we previously used this proc number.
- */
- memset(&shstatent->stats, 0, sizeof(shstatent->stats));
- pgstat_unlock_entry(entry_ref);
-}
-
-/*
- * Backend statistics are not collected for all BackendTypes.
- *
- * The following BackendTypes do not participate in the backend stats
- * subsystem:
- * - The same and for the same reasons as in pgstat_tracks_io_bktype().
- * - B_BG_WRITER, B_CHECKPOINTER, B_STARTUP and B_AUTOVAC_LAUNCHER because their
- * I/O stats are already visible in pg_stat_io and there is only one of those.
- *
- * Function returns true if BackendType participates in the backend stats
- * subsystem and false if it does not.
- *
- * When adding a new BackendType, also consider adding relevant restrictions to
- * pgstat_tracks_io_object() and pgstat_tracks_io_op().
- */
-bool
-pgstat_tracks_backend_bktype(BackendType bktype)
-{
- /*
- * List every type so that new backend types trigger a warning about
- * needing to adjust this switch.
- */
- switch (bktype)
- {
- case B_INVALID:
- case B_AUTOVAC_LAUNCHER:
- case B_DEAD_END_BACKEND:
- case B_ARCHIVER:
- case B_LOGGER:
- case B_BG_WRITER:
- case B_CHECKPOINTER:
- case B_IO_WORKER:
- case B_STARTUP:
- case B_DATACHECKSUMSWORKER_LAUNCHER:
- case B_DATACHECKSUMSWORKER_WORKER:
- return false;
-
- case B_AUTOVAC_WORKER:
- case B_BACKEND:
- case B_BG_WORKER:
- case B_STANDALONE_BACKEND:
- case B_SLOTSYNC_WORKER:
- case B_WAL_RECEIVER:
- case B_WAL_SENDER:
- case B_WAL_SUMMARIZER:
- case B_WAL_WRITER:
- return true;
- }
-
- return false;
-}
-
-void
-pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts)
-{
- ((PgStatShared_Backend *) header)->stats.stat_reset_timestamp = ts;
-}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index cd6bfc9a33b..8282a8413ef 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -534,15 +534,6 @@ typedef struct PgStat_WalStats
TimestampTz stat_reset_timestamp;
} PgStat_WalStats;
-/* -------
- * PgStat_Backend Backend statistics
- * -------
- */
-typedef struct PgStat_Backend
-{
- TimestampTz stat_reset_timestamp;
-} PgStat_Backend;
-
/*
* Functions in pgstat.c
*/
@@ -579,16 +570,6 @@ extern bool pgstat_have_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_report_archiver(const char *xlog, bool failed);
extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void);
-/*
- * Functions in pgstat_backend.c
- */
-
-extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber);
-extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid,
- BackendType *bktype);
-extern bool pgstat_tracks_backend_bktype(BackendType bktype);
-extern void pgstat_create_backend(ProcNumber procnum);
-
/*
* Functions in pgstat_bgwriter.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 95d52e1a26e..78471b05773 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -601,12 +601,6 @@ typedef struct PgStatShared_ReplSlot
PgStat_StatReplSlotEntry stats;
} PgStatShared_ReplSlot;
-typedef struct PgStatShared_Backend
-{
- PgStatShared_Common header;
- PgStat_Backend stats;
-} PgStatShared_Backend;
-
/*
* Central shared memory entry for the cumulative stats system.
*
@@ -789,13 +783,6 @@ extern void pgstat_archiver_init_shmem_cb(void *stats);
extern void pgstat_archiver_reset_all_cb(TimestampTz ts);
extern void pgstat_archiver_snapshot_cb(void);
-/*
- * Functions in pgstat_backend.c
- */
-
-extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
- TimestampTz ts);
-
/*
* Functions in pgstat_bgwriter.c
*/
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index 45ca599d0dd..489e8fa38bf 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -30,16 +30,15 @@
#define PGSTAT_KIND_FUNCTION 4 /* per-function statistics */
#define PGSTAT_KIND_REPLSLOT 5 /* per-slot statistics */
#define PGSTAT_KIND_SUBSCRIPTION 6 /* per-subscription statistics */
-#define PGSTAT_KIND_BACKEND 7 /* per-backend statistics */
/* stats for fixed-numbered objects */
-#define PGSTAT_KIND_ARCHIVER 8
-#define PGSTAT_KIND_BGWRITER 9
-#define PGSTAT_KIND_CHECKPOINTER 10
-#define PGSTAT_KIND_IO 11
-#define PGSTAT_KIND_LOCK 12
-#define PGSTAT_KIND_SLRU 13
-#define PGSTAT_KIND_WAL 14
+#define PGSTAT_KIND_ARCHIVER 7
+#define PGSTAT_KIND_BGWRITER 8
+#define PGSTAT_KIND_CHECKPOINTER 9
+#define PGSTAT_KIND_IO 10
+#define PGSTAT_KIND_LOCK 11
+#define PGSTAT_KIND_SLRU 12
+#define PGSTAT_KIND_WAL 13
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 4255c1234f4..a53daf1338e 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -120,15 +120,14 @@ SELECT id, name, fixed_amount,
4 | function | f | f | t
5 | replslot | f | t | t
6 | subscription | f | t | t
- 7 | backend | f | t | f
- 8 | archiver | t | f | t
- 9 | bgwriter | t | f | t
- 10 | checkpointer | t | f | t
- 11 | io | t | f | t
- 12 | lock | t | f | t
- 13 | slru | t | f | t
- 14 | wal | t | f | t
-(14 rows)
+ 7 | archiver | t | f | t
+ 8 | bgwriter | t | f | t
+ 9 | checkpointer | t | f | t
+ 10 | io | t | f | t
+ 11 | lock | t | f | t
+ 12 | slru | t | f | t
+ 13 | wal | t | f | t
+(13 rows)
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 48a1deb9bca..5db53416c7e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2316,7 +2316,6 @@ PgFdwSamplingMethod
PgFdwScanState
PgIfAddrCallback
PgStatShared_Archiver
-PgStatShared_Backend
PgStatShared_BgWriter
PgStatShared_Checkpointer
PgStatShared_Common
@@ -2338,7 +2337,6 @@ PgStatShared_Subscription
PgStatShared_Wal
PgStatShared_WalBackendEntry
PgStat_ArchiverStats
-PgStat_Backend
PgStat_BackendIO
PgStat_BackendSubEntry
PgStat_BgWriterStats
--
2.34.1
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-10 14:45 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-13 15:41 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-09-21 07:15 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 09:17 ` Re: Redesign per-backend statistics Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 11+ messages in thread
From: Bertrand Drouvot @ 2026-09-21 07:15 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers@lists.postgresql.org, Andres Freund <andres@anarazel.de>; Sami Imseih <samimseih@gmail.com>
Hi,
On Thu, Aug 13, 2026 at 03:41:58PM +0000, Bertrand Drouvot wrote:
> Hi,
>
> On Mon, Aug 10, 2026 at 02:45:16PM +0000, Bertrand Drouvot wrote:
> > Given that the cost is still sub millisecond at hundreds of connections and a few
> > milliseconds at 10000 and given the impact mentioned above, I don't think this is
> > a practical blocker though. What do you think?
> >
>
> A rebase was needed, so while at it, I implemented what we have discussed so far,
> means:
>
> - moved the generic per-backend statistics infrastructure to a new pgstat_per_backend.c
> file
> - split the previous 0005 into the IO migration and the removal of PGSTAT_KIND_BACKEND
> and pgstat_backend.c.
PFA another mandatory rebase (tiny change to test_misc/meson.build).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v3-0001-pgstat-add-tests-for-per-backend-statistics.patch (21.2K, ../../arDZoI%2F5xHU2utRh@bdtpg/2-v3-0001-pgstat-add-tests-for-per-backend-statistics.patch)
download | inline diff:
From fa05163827d350487b87354c5ec562092e8d2be9 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:23:40 +0000
Subject: [PATCH v3 1/6] pgstat: add tests for per-backend statistics
Add an isolation test for per-backend WAL statistics. Verify that SNAPSHOT, CACHE
and NONE modes behave as expected.
Add a TAP test that starts a backend after another session has built a
statistics snapshot. Per-backend WAL, IO, and Lock queries for the new
backend must return no rows, rather than fabricated rows containing zeroes.
Extend test_shm_mq with a shared-memory-only worker that reports WAL usage
through a routine nonblocking flush and remains alive while the controlling
backend verifies that the global WAL counters include it.
Finally, extend the core statistics regression test for per-backend WAL
reset. Check that resetting one backend reduces its counters and sets its reset
timestamp without removing the same activity from the global WAL statistics.
These tests will serve as compatibility coverage for the per-backend redesign
changes coming in following commits.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
.../isolation/expected/stats-per-backend.out | 143 ++++++++++++++++++
src/test/isolation/isolation_schedule | 1 +
.../isolation/specs/stats-per-backend.spec | 123 +++++++++++++++
src/test/modules/test_misc/meson.build | 1 +
.../modules/test_misc/t/016_stats_snapshot.pl | 48 ++++++
.../test_shm_mq/expected/test_shm_mq.out | 7 +
.../modules/test_shm_mq/sql/test_shm_mq.sql | 3 +
src/test/modules/test_shm_mq/test.c | 51 +++++++
.../modules/test_shm_mq/test_shm_mq--1.0.sql | 4 +
src/test/modules/test_shm_mq/test_shm_mq.h | 4 +
src/test/modules/test_shm_mq/worker.c | 10 ++
src/test/regress/expected/stats.out | 29 ++++
src/test/regress/sql/stats.sql | 10 ++
13 files changed, 434 insertions(+)
30.6% src/test/isolation/expected/
31.4% src/test/isolation/specs/
9.7% src/test/modules/test_misc/t/
17.6% src/test/modules/test_shm_mq/
5.6% src/test/regress/expected/
4.3% src/test/regress/sql/
diff --git a/src/test/isolation/expected/stats-per-backend.out b/src/test/isolation/expected/stats-per-backend.out
new file mode 100644
index 00000000000..0d85ae99033
--- /dev/null
+++ b/src/test/isolation/expected/stats-per-backend.out
@@ -0,0 +1,143 @@
+Parsed test spec with 2 sessions
+
+starting permutation: s1_fetch_consistency_snapshot s1_begin s1_build_snapshot s2_generate s1_check_snapshot_backend s1_commit s1_check_live_backend
+step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot';
+step s1_begin: BEGIN;
+step s1_build_snapshot:
+ SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal;
+
+snapshot_created
+----------------
+t
+(1 row)
+
+step s2_generate:
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_check_snapshot_backend:
+ SELECT wal_records = 0 AS snapshot_excludes_later_update
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+snapshot_excludes_later_update
+------------------------------
+t
+(1 row)
+
+step s1_commit: COMMIT;
+step s1_check_live_backend:
+ SELECT wal_records > 0 AS live_entry_exists
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+live_entry_exists
+-----------------
+t
+(1 row)
+
+
+starting permutation: s1_fetch_consistency_cache s2_generate s1_begin s1_save_backend_stats s2_generate_more s1_check_cached_backend s1_commit s1_check_refreshed_backend
+step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache';
+step s2_generate:
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_begin: BEGIN;
+step s1_save_backend_stats:
+ INSERT INTO stats_per_backend_saved
+ SELECT wal_records
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+step s2_generate_more:
+ INSERT INTO stats_per_backend_data VALUES (2);
+ SELECT pg_stat_force_next_flush();
+
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s1_check_cached_backend:
+ SELECT current.wal_records = saved.wal_records AS cache_is_stable
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+
+cache_is_stable
+---------------
+t
+(1 row)
+
+step s1_commit: COMMIT;
+step s1_check_refreshed_backend:
+ SELECT current.wal_records > saved.wal_records AS cache_is_refreshed
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+
+cache_is_refreshed
+------------------
+t
+(1 row)
+
+
+starting permutation: s1_reset_backend s1_check_backend_reset
+step s1_reset_backend:
+ SELECT pg_stat_reset_backend_stats(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+pg_stat_reset_backend_stats
+---------------------------
+
+(1 row)
+
+step s1_check_backend_reset:
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+reset_timestamp_set
+-------------------
+t
+(1 row)
+
+
+starting permutation: s1_reset_shared s1_check_shared_reset
+step s1_reset_shared:
+ SELECT pg_stat_reset_shared('wal');
+
+pg_stat_reset_shared
+--------------------
+
+(1 row)
+
+step s1_check_shared_reset:
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+
+reset_timestamp_set
+-------------------
+t
+(1 row)
+
diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule
index 8470d50d2bc..21351401190 100644
--- a/src/test/isolation/isolation_schedule
+++ b/src/test/isolation/isolation_schedule
@@ -107,6 +107,7 @@ test: vacuum-concurrent-drop
test: vacuum-conflict
test: vacuum-skip-locked
test: stats
+test: stats-per-backend
test: horizons
test: predicate-hash
test: predicate-gist
diff --git a/src/test/isolation/specs/stats-per-backend.spec b/src/test/isolation/specs/stats-per-backend.spec
new file mode 100644
index 00000000000..5dfc8dbe90e
--- /dev/null
+++ b/src/test/isolation/specs/stats-per-backend.spec
@@ -0,0 +1,123 @@
+# Test snapshot, cache, and reset behavior for per-backend statistics.
+#
+# Session s2 generates and flushes WAL records, which provide a deterministic
+# per-backend counter. Session s1 verifies that SNAPSHOT mode fixes all
+# per-backend entries at the initial statistics snapshot, while CACHE mode
+# fixes each entry on first access and refreshes it after the transaction
+# ends. The final permutations verify reset timestamps after per-backend and
+# shared resets.
+
+setup
+{
+ CREATE TABLE stats_per_backend_data(id int);
+ CREATE TABLE stats_per_backend_saved(wal_records bigint);
+}
+
+teardown
+{
+ DROP TABLE stats_per_backend_data;
+ DROP TABLE stats_per_backend_saved;
+}
+
+session s1
+setup { SET stats_fetch_consistency = 'none'; }
+
+step s1_fetch_consistency_cache { SET stats_fetch_consistency = 'cache'; }
+step s1_fetch_consistency_snapshot { SET stats_fetch_consistency = 'snapshot'; }
+step s1_begin { BEGIN; }
+step s1_commit { COMMIT; }
+step s1_build_snapshot {
+ SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal;
+}
+step s1_check_snapshot_backend {
+ SELECT wal_records = 0 AS snapshot_excludes_later_update
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_live_backend {
+ SELECT wal_records > 0 AS live_entry_exists
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_save_backend_stats {
+ INSERT INTO stats_per_backend_saved
+ SELECT wal_records
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_cached_backend {
+ SELECT current.wal_records = saved.wal_records AS cache_is_stable
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+}
+step s1_check_refreshed_backend {
+ SELECT current.wal_records > saved.wal_records AS cache_is_refreshed
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2')) AS current
+ CROSS JOIN stats_per_backend_saved AS saved;
+}
+step s1_reset_backend {
+ SELECT pg_stat_reset_backend_stats(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_check_backend_reset {
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+step s1_reset_shared {
+ SELECT pg_stat_reset_shared('wal');
+}
+step s1_check_shared_reset {
+ SELECT stats_reset IS NOT NULL AS reset_timestamp_set
+ FROM pg_stat_get_backend_wal(
+ (SELECT pid FROM pg_stat_activity
+ WHERE application_name = 'isolation/stats-per-backend/s2'));
+}
+
+session s2
+step s2_generate {
+ INSERT INTO stats_per_backend_data VALUES (1);
+ SELECT pg_stat_force_next_flush();
+}
+step s2_generate_more {
+ INSERT INTO stats_per_backend_data VALUES (2);
+ SELECT pg_stat_force_next_flush();
+}
+
+# with stats_fetch_consistency=snapshot s1 should not see flushed changes from
+# s2 after building the statistics snapshot, but should see them after commit
+permutation
+ s1_fetch_consistency_snapshot
+ s1_begin
+ s1_build_snapshot
+ s2_generate
+ s1_check_snapshot_backend
+ s1_commit
+ s1_check_live_backend
+
+# with stats_fetch_consistency=cache s1 should not see flushed changes from s2
+# after the first access, but should see them after commit
+permutation
+ s1_fetch_consistency_cache
+ s2_generate
+ s1_begin
+ s1_save_backend_stats
+ s2_generate_more
+ s1_check_cached_backend
+ s1_commit
+ s1_check_refreshed_backend
+
+# a per-backend reset should set that backend's reset timestamp
+permutation s1_reset_backend s1_check_backend_reset
+
+# a shared WAL reset should set the reset timestamp for live backends
+permutation s1_reset_shared s1_check_shared_reset
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index 5d81f5b13be..9521b25172f 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -24,6 +24,7 @@ tests += {
't/013_temp_obj_multisession.pl',
't/014_log_statement_max_length.pl',
't/015_temp_schema_exit_deferrable.pl',
+ 't/016_stats_snapshot.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/016_stats_snapshot.pl b/src/test/modules/test_misc/t/016_stats_snapshot.pl
new file mode 100644
index 00000000000..43c9157bbb4
--- /dev/null
+++ b/src/test/modules/test_misc/t/016_stats_snapshot.pl
@@ -0,0 +1,48 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Verify that a statistics snapshot excludes backends that started after the
+# snapshot was created.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('stats_snapshot');
+$node->init;
+$node->start;
+
+my $snapshot = $node->background_psql('postgres');
+
+$snapshot->query_safe(
+ q[
+ BEGIN;
+ SET LOCAL stats_fetch_consistency = 'snapshot';
+ SELECT count(*) FROM pg_stat_wal;
+]);
+
+# Establish this connection only after the first session built its snapshot.
+my $late = $node->background_psql('postgres');
+my $late_pid = $late->query_safe('SELECT pg_backend_pid()');
+
+# WAL returns one scalar composite, so count a field to distinguish a NULL
+# result from the fabricated all-zero row.
+is( $snapshot->query_safe(
+ qq[
+ SELECT
+ (SELECT count(wal_records)
+ FROM pg_stat_get_backend_wal($late_pid)),
+ (SELECT count(*) FROM pg_stat_get_backend_io($late_pid)),
+ (SELECT count(*) FROM pg_stat_get_backend_lock($late_pid));
+ ]),
+ '0|0|0',
+ 'statistics snapshot excludes a backend created later');
+
+$snapshot->query_safe('ROLLBACK');
+$late->quit;
+$snapshot->quit;
+$node->stop;
+
+done_testing();
diff --git a/src/test/modules/test_shm_mq/expected/test_shm_mq.out b/src/test/modules/test_shm_mq/expected/test_shm_mq.out
index c4858b0c205..ec3a7d3199e 100644
--- a/src/test/modules/test_shm_mq/expected/test_shm_mq.out
+++ b/src/test/modules/test_shm_mq/expected/test_shm_mq.out
@@ -34,3 +34,10 @@ SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int
(1 row)
+-- A shmem-only worker can publish statistics before its clean exit.
+SELECT test_shm_mq_worker_stats();
+ test_shm_mq_worker_stats
+--------------------------
+ t
+(1 row)
+
diff --git a/src/test/modules/test_shm_mq/sql/test_shm_mq.sql b/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
index 9de19d304a2..e80e4399881 100644
--- a/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
+++ b/src/test/modules/test_shm_mq/sql/test_shm_mq.sql
@@ -10,3 +10,6 @@ SELECT test_shm_mq(1024, 'a', 2001, 1);
SELECT test_shm_mq(32768, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,(100+900*random())::int)), 10000, 1);
SELECT test_shm_mq(100, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,(100+200*random())::int)), 10000, 1);
SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,270000)), 200, 3);
+
+-- A shmem-only worker can publish statistics before its clean exit.
+SELECT test_shm_mq_worker_stats();
diff --git a/src/test/modules/test_shm_mq/test.c b/src/test/modules/test_shm_mq/test.c
index 0e55287e510..4464d796ef9 100644
--- a/src/test/modules/test_shm_mq/test.c
+++ b/src/test/modules/test_shm_mq/test.c
@@ -26,6 +26,7 @@ PG_MODULE_MAGIC;
PG_FUNCTION_INFO_V1(test_shm_mq);
PG_FUNCTION_INFO_V1(test_shm_mq_pipelined);
+PG_FUNCTION_INFO_V1(test_shm_mq_worker_stats);
static void verify_message(Size origlen, char *origdata, Size newlen,
char *newdata);
@@ -253,6 +254,56 @@ test_shm_mq_pipelined(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+/*
+ * Verify that a shmem-only worker can publish statistics through a routine
+ * nonblocking flush while it remains alive.
+ */
+Datum
+test_shm_mq_worker_stats(PG_FUNCTION_ARGS)
+{
+ const char *message = PG_TEST_SHM_MQ_STATS_MESSAGE;
+ PgStat_Counter before;
+ PgStat_Counter after;
+ dsm_segment *seg;
+ shm_mq_handle *outqh;
+ shm_mq_handle *inqh;
+ shm_mq_result res;
+ Size len;
+ void *data;
+
+ pgstat_clear_snapshot();
+ before = pgstat_fetch_stat_wal()->wal_counters.wal_records;
+
+ test_shm_mq_setup(1024, 1, &seg, &outqh, &inqh);
+
+ res = shm_mq_send(outqh, strlen(message), message, false, true);
+
+ if (res != SHM_MQ_SUCCESS)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not send statistics test message")));
+
+ res = shm_mq_receive(inqh, &len, &data, false);
+
+ if (res != SHM_MQ_SUCCESS)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not receive statistics test message")));
+
+ verify_message(strlen(message), (char *) message, len, data);
+
+ /*
+ * The worker is waiting for another message here. Check its live entry
+ * before detaching the queues and triggering its shutdown flush.
+ */
+ pgstat_clear_snapshot();
+ after = pgstat_fetch_stat_wal()->wal_counters.wal_records;
+
+ dsm_detach(seg);
+
+ PG_RETURN_BOOL(after >= before + PG_TEST_SHM_MQ_STATS_RECORDS);
+}
+
/*
* Verify that two messages are the same.
*/
diff --git a/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql b/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
index 56db05d93df..dce6f7378a5 100644
--- a/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
+++ b/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql
@@ -17,3 +17,7 @@ CREATE FUNCTION test_shm_mq_pipelined(queue_size pg_catalog.int8,
verify pg_catalog.bool default true)
RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION test_shm_mq_worker_stats()
+ RETURNS pg_catalog.bool
+ AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_shm_mq/test_shm_mq.h b/src/test/modules/test_shm_mq/test_shm_mq.h
index b6a0290289c..501be180642 100644
--- a/src/test/modules/test_shm_mq/test_shm_mq.h
+++ b/src/test/modules/test_shm_mq/test_shm_mq.h
@@ -21,6 +21,10 @@
/* Identifier for shared memory segments used by this extension. */
#define PG_TEST_SHM_MQ_MAGIC 0x79fb2447
+/* Special message used to test statistics from a shmem-only worker. */
+#define PG_TEST_SHM_MQ_STATS_MESSAGE "test_shm_mq worker stats"
+#define PG_TEST_SHM_MQ_STATS_RECORDS 1000000
+
/*
* This structure is stored in the dynamic shared memory segment. We use
* it to determine whether all workers started up OK and successfully
diff --git a/src/test/modules/test_shm_mq/worker.c b/src/test/modules/test_shm_mq/worker.c
index 0ba1cfcac47..afcea7fc0cb 100644
--- a/src/test/modules/test_shm_mq/worker.c
+++ b/src/test/modules/test_shm_mq/worker.c
@@ -19,6 +19,7 @@
#include "postgres.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -27,6 +28,7 @@
#include "storage/shm_mq.h"
#include "storage/shm_toc.h"
#include "tcop/tcopprot.h"
+#include "utils/pgstat_internal.h"
#include "test_shm_mq.h"
@@ -185,6 +187,14 @@ copy_messages(shm_mq_handle *inqh, shm_mq_handle *outqh)
if (res != SHM_MQ_SUCCESS)
break;
+ if (len == strlen(PG_TEST_SHM_MQ_STATS_MESSAGE) &&
+ memcmp(data, PG_TEST_SHM_MQ_STATS_MESSAGE, len) == 0)
+ {
+ pgWalUsage.wal_records += PG_TEST_SHM_MQ_STATS_RECORDS;
+ pgstat_report_fixed = true;
+ (void) pgstat_report_stat(false);
+ }
+
/* Send it back out. */
res = shm_mq_send(outqh, len, data, false, true);
if (res != SHM_MQ_SUCCESS)
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 8b15471248b..e3dd0ce89b0 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1130,6 +1130,35 @@ SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_bac
t
(1 row)
+-- Test pg_stat_reset_backend_stats() for WAL
+SELECT wal_bytes AS wal_bytes_before_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT pg_stat_reset_backend_stats(pg_backend_pid());
+ pg_stat_reset_backend_stats
+-----------------------------
+
+(1 row)
+
+SELECT wal_bytes AS wal_bytes_after_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT :wal_bytes_after_reset < :wal_bytes_before_reset;
+ ?column?
+----------
+ t
+(1 row)
+
+-- global view should still have the data
+SELECT wal_bytes >= :wal_bytes_before_reset FROM pg_stat_wal;
+ ?column?
+----------
+ t
+(1 row)
+
+-- reset timestamp should be set
+SELECT stats_reset IS NOT NULL FROM pg_stat_get_backend_wal(pg_backend_pid());
+ ?column?
+----------
+ t
+(1 row)
+
-- Test pg_stat_get_backend_idset() and some allied functions.
-- In particular, verify that their notion of backend ID matches
-- our temp schema index.
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index 674637e172b..f4931fd84de 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -528,6 +528,16 @@ SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal;
SELECT pg_stat_force_next_flush();
SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_backend_pid());
+-- Test pg_stat_reset_backend_stats() for WAL
+SELECT wal_bytes AS wal_bytes_before_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT pg_stat_reset_backend_stats(pg_backend_pid());
+SELECT wal_bytes AS wal_bytes_after_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset
+SELECT :wal_bytes_after_reset < :wal_bytes_before_reset;
+-- global view should still have the data
+SELECT wal_bytes >= :wal_bytes_before_reset FROM pg_stat_wal;
+-- reset timestamp should be set
+SELECT stats_reset IS NOT NULL FROM pg_stat_get_backend_wal(pg_backend_pid());
+
-- Test pg_stat_get_backend_idset() and some allied functions.
-- In particular, verify that their notion of backend ID matches
-- our temp schema index.
--
2.34.1
[text/x-diff] v3-0002-pgstat-add-new-infrastructure-for-per-backend-sta.patch (25.4K, ../../arDZoI%2F5xHU2utRh@bdtpg/3-v3-0002-pgstat-add-new-infrastructure-for-per-backend-sta.patch)
download | inline diff:
From 94c8483511c29e5837fbd299294ecb42d8cac265 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 29 Jul 2026 14:11:27 +0000
Subject: [PATCH v3 2/6] pgstat: add new infrastructure for per-backend
statistics
Add a new infrastructure for per-backend statistics that keep one shared entry
per live backend while keeping their existing shared data as global stats for
exited backends.
Define a common dshash entry header containing the ProcNumber key, BackendType,
and content LWLock. Extend PgStat_KindInfo with per-backend statistics related
informations.
At shared-memory initialization, create a dshash for every participating kind.
Create entries during pgstat_initialize(), so that auxiliary and shared memory
only processes are covered. Nonblocking flushes can then acquire the cached
entry's content lock without performing a dshash lookup, DSA address resolution,
allocation, or hash resize.
Add helpers to fetch per-backend entries, include live entries in global
snapshots, and transfer entries to the global statistics when backends exit.
Add a local cache keyed by statistics kind and ProcNumber for per-backend
fetches.
No statistics kind registers per backend metadata in this commit, so the new
shared memory creation and backend initialization loops are no ops. Subsequent
commits will add WAL, Lock, and IO statistics into the infrastructure individually.
Limit this infrastructure to built in fixed numbered statistics kinds as this is
the only use case we have had so far. We could extend to variable ones later on
if needed.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
src/backend/utils/activity/Makefile | 1 +
src/backend/utils/activity/meson.build | 1 +
src/backend/utils/activity/pgstat.c | 21 +
.../utils/activity/pgstat_per_backend.c | 513 ++++++++++++++++++
src/backend/utils/activity/pgstat_shmem.c | 46 ++
src/include/utils/pgstat_internal.h | 45 ++
src/tools/pgindent/typedefs.list | 4 +
7 files changed, 631 insertions(+)
89.6% src/backend/utils/activity/
9.7% src/include/utils/
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index 2e32d1485d6..8924b10ed2e 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -29,6 +29,7 @@ OBJS = \
pgstat_io.o \
pgstat_kind.o \
pgstat_lock.o \
+ pgstat_per_backend.o \
pgstat_relation.o \
pgstat_replslot.o \
pgstat_shmem.o \
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index e6dcb2e26fc..6434c377e53 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -14,6 +14,7 @@ backend_sources += files(
'pgstat_io.c',
'pgstat_kind.c',
'pgstat_lock.c',
+ 'pgstat_per_backend.c',
'pgstat_relation.c',
'pgstat_replslot.c',
'pgstat_shmem.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 5177f880f70..99ac24f166d 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -70,6 +70,7 @@
* To keep things manageable, stats handling is split across several
* files. Infrastructure pieces are in:
* - pgstat.c - this file, to tie it all together
+ * - pgstat_per_backend.c - generic per-backend statistics infrastructure
* - pgstat_shmem.c - nearly everything dealing with shared memory, including
* the maintenance of hashtable entries
* - pgstat_xact.c - transactional integration, including the transactional
@@ -689,6 +690,13 @@ pgstat_initialize(void)
pgstat_attach_shmem();
+ /*
+ * Create and cache per-backend statistics entries here. This also covers
+ * processes that never call InitPostgres(), such as shared-memory-only
+ * background workers.
+ */
+ pgstat_create_my_per_backend_entries();
+
pgstat_init_snapshot_fixed();
/* Backend initialization callbacks */
@@ -962,6 +970,7 @@ pgstat_clear_snapshot(void)
/* Reset variables */
pgStatLocal.snapshot.context = NULL;
+ pgStatLocal.snapshot.per_backend_stats = NULL;
}
/*
@@ -975,6 +984,13 @@ pgstat_clear_snapshot(void)
force_stats_snapshot_clear = false;
}
+void
+pgstat_maybe_clear_snapshot(void)
+{
+ if (force_stats_snapshot_clear)
+ pgstat_clear_snapshot();
+}
+
void *
pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *may_free)
{
@@ -1556,6 +1572,11 @@ pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
errdetail("Custom cumulative statistics must be registered while initializing modules in \"%s\".",
"shared_preload_libraries")));
+ if (kind_info->per_backend_data_len != 0)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Per-backend statistics are not supported for custom cumulative statistics.")));
+
/*
* Check some data for fixed-numbered stats.
*/
diff --git a/src/backend/utils/activity/pgstat_per_backend.c b/src/backend/utils/activity/pgstat_per_backend.c
new file mode 100644
index 00000000000..b82787e9ea5
--- /dev/null
+++ b/src/backend/utils/activity/pgstat_per_backend.c
@@ -0,0 +1,513 @@
+/* -------------------------------------------------------------------------
+ *
+ * pgstat_per_backend.c
+ * Generic infrastructure for per-backend statistics.
+ *
+ * This file manages the dedicated per-kind dshashes used for per-backend
+ * statistics, including entry creation, fetching, snapshots, transfer to
+ * global statistics, and removal.
+ *
+ * Copyright (c) 2001-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/activity/pgstat_per_backend.c
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/dshash.h"
+#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+
+/* hash table for per-backend stats snapshot entries */
+typedef struct PgStat_PerBackendSnapshotKey
+{
+ PgStat_Kind kind;
+ ProcNumber procnum;
+} PgStat_PerBackendSnapshotKey;
+
+typedef struct PgStat_PerBackendSnapshotEntry
+{
+ PgStat_PerBackendSnapshotKey key;
+ char status; /* for simplehash use */
+ void *data; /* the stats data itself */
+} PgStat_PerBackendSnapshotEntry;
+
+#define SH_PREFIX pgstat_per_backend_snapshot
+#define SH_ELEMENT_TYPE PgStat_PerBackendSnapshotEntry
+#define SH_KEY_TYPE PgStat_PerBackendSnapshotKey
+#define SH_KEY key
+#define SH_HASH_KEY(tb, key) \
+ fasthash32((const char *) &key, sizeof(PgStat_PerBackendSnapshotKey), 0)
+#define SH_EQUAL(tb, a, b) \
+ (memcmp(&a, &b, sizeof(PgStat_PerBackendSnapshotKey)) == 0)
+#define SH_SCOPE static inline
+#define SH_DEFINE
+#define SH_DECLARE
+#include "lib/simplehash.h"
+
+/* Per-kind, backend-local state for the per-backend dshashes. */
+typedef struct PgStat_PerBackendLocalState
+{
+ dshash_table *hash;
+ PgStatShared_PerBackendEntry *my_entry;
+} PgStat_PerBackendLocalState;
+
+static PgStat_PerBackendLocalState per_backend_states[PGSTAT_KIND_BUILTIN_SIZE];
+
+/*
+ * Look up a per-backend stats entry in the backend snapshot hash.
+ *
+ * The returned entry may be empty when no matching statistics were found on
+ * first access.
+ */
+static PgStat_PerBackendSnapshotEntry *
+pgstat_lookup_per_backend_entry(PgStat_Kind kind, ProcNumber procnum)
+{
+ PgStat_PerBackendSnapshotKey key;
+
+ if (pgStatLocal.snapshot.per_backend_stats == NULL)
+ return NULL;
+
+ key.kind = kind;
+ key.procnum = procnum;
+
+ return pgstat_per_backend_snapshot_lookup(pgStatLocal.snapshot.per_backend_stats,
+ key);
+}
+
+/*
+ * Cache a per-backend stats entry in the backend snapshot hash.
+ *
+ * If data is NULL, cache an empty entry to record that no matching statistics
+ * were found on first access.
+ */
+static void *
+pgstat_cache_per_backend_entry(PgStat_Kind kind, ProcNumber procnum,
+ const void *data)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendSnapshotKey key;
+ PgStat_PerBackendSnapshotEntry *entry;
+ bool found;
+
+ Assert(pgstat_fetch_consistency > PGSTAT_FETCH_CONSISTENCY_NONE);
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ /* Ensure snapshot context exists */
+ if (!pgStatLocal.snapshot.context)
+ pgStatLocal.snapshot.context = AllocSetContextCreate(TopMemoryContext,
+ "PgStat Snapshot",
+ ALLOCSET_SMALL_SIZES);
+
+ /* Create per-backend hash on first use */
+ if (pgStatLocal.snapshot.per_backend_stats == NULL)
+ pgStatLocal.snapshot.per_backend_stats =
+ pgstat_per_backend_snapshot_create(pgStatLocal.snapshot.context, 64, NULL);
+
+ key.kind = kind;
+ key.procnum = procnum;
+
+ /* If already cached, return cached data */
+ entry = pgstat_per_backend_snapshot_lookup(pgStatLocal.snapshot.per_backend_stats, key);
+
+ if (entry)
+ return entry->data;
+
+ /* Insert new entry into the hash */
+ entry = pgstat_per_backend_snapshot_insert(pgStatLocal.snapshot.per_backend_stats,
+ key, &found);
+ Assert(!found);
+
+ if (data != NULL)
+ {
+ entry->data = MemoryContextAlloc(pgStatLocal.snapshot.context,
+ kind_info->per_backend_data_len);
+ memcpy(entry->data, data, kind_info->per_backend_data_len);
+ }
+ else
+ entry->data = NULL;
+
+ return entry->data;
+}
+
+static inline PgStat_PerBackendLocalState *
+pgstat_get_per_backend_local_state(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info PG_USED_FOR_ASSERTS_ONLY = pgstat_get_kind_info(kind);
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->fixed_amount);
+
+ if (!pgstat_is_kind_builtin(kind))
+ elog(ERROR, "invalid statistics kind: %u", kind);
+
+ return &per_backend_states[kind];
+}
+
+/*
+ * Create and cache this process's entry for one per-backend statistics kind.
+ */
+static PgStatShared_PerBackendEntry *
+pgstat_create_my_per_backend_entry(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ dshash_table *hash = pgstat_per_backend_attach(kind);
+ PgStatShared_PerBackendEntry *entry;
+ bool found;
+
+ Assert(state->my_entry == NULL);
+
+ if (hash == NULL)
+ return NULL;
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ entry = dshash_find_or_insert(hash, &MyProcNumber, &found);
+
+ /*
+ * A forced flush during early backend startup may already have created
+ * the entry. Preserve any statistics it contains.
+ */
+ if (!found)
+ {
+ entry->backend_type = MyBackendType;
+ LWLockInitialize(&entry->lock, LWTRANCHE_PGSTATS_DATA);
+ memset((char *) entry + kind_info->per_backend_data_off, 0,
+ kind_info->per_backend_data_len);
+ }
+
+ state->my_entry = entry;
+ dshash_release_lock(hash, entry);
+
+ return entry;
+}
+
+/*
+ * Create entries for all the kinds that use per-backend dshashes.
+ *
+ * Allocations and dshash resizes are deliberately done during backend
+ * initialization so routine stats flushes only need to conditionally acquire
+ * the cached entry's content lock.
+ */
+void
+pgstat_create_my_per_backend_entries(void)
+{
+ for (PgStat_Kind kind = PGSTAT_KIND_BUILTIN_MIN;
+ kind <= PGSTAT_KIND_BUILTIN_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (kind_info == NULL || kind_info->per_backend_data_len == 0)
+ continue;
+
+ (void) pgstat_create_my_per_backend_entry(kind);
+ }
+}
+
+/*
+ * Attach to the per-backend dshash for the given kind.
+ * Returns NULL if the hash is not available (e.g. during bootstrap or
+ * if this kind doesn't have per-backend tracking).
+ */
+dshash_table *
+pgstat_per_backend_attach(PgStat_Kind kind)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table_handle *handle_ptr;
+ MemoryContext oldcontext;
+ dshash_parameters params;
+
+ if (state->hash != NULL)
+ return state->hash;
+
+ if (!kind_info || !kind_info->per_backend_data_len)
+ return NULL;
+
+ /* Get the shared struct for this kind */
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ handle_ptr = (dshash_table_handle *) (shared_struct + kind_info->per_backend_hash_handle_off);
+
+ if (*handle_ptr == DSHASH_HANDLE_INVALID)
+ return NULL;
+
+ /*
+ * Build dshash parameters from kind info. All per-backend hashes use
+ * ProcNumber keys and dshash_memcmp/dshash_memhash.
+ */
+ params.key_size = sizeof(ProcNumber);
+ params.entry_size = kind_info->per_backend_data_off + kind_info->per_backend_data_len;
+ params.compare_function = dshash_memcmp;
+ params.hash_function = dshash_memhash;
+ params.copy_function = dshash_memcpy;
+ params.tranche_id = LWTRANCHE_PGSTATS_HASH;
+
+ /* Attach in TopMemoryContext */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+ state->hash = dshash_attach(pgStatLocal.dsa, ¶ms, *handle_ptr, NULL);
+ MemoryContextSwitchTo(oldcontext);
+
+ return state->hash;
+}
+
+/*
+ * Lock this process's cached entry for a per-backend statistics kind.
+ *
+ * A missing entry is only created in the blocking path. The routine nowait
+ * path must not perform dshash lookups, allocations, or DSA address
+ * resolution.
+ */
+void *
+pgstat_lock_my_per_backend_entry(PgStat_Kind kind, bool nowait)
+{
+ const PgStat_KindInfo *kind_info PG_USED_FOR_ASSERTS_ONLY = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ PgStatShared_PerBackendEntry *entry = state->my_entry;
+
+ Assert(kind_info != NULL);
+ Assert(kind_info->per_backend_data_len > 0);
+
+ if (entry == NULL)
+ {
+ if (nowait)
+ return NULL;
+
+ entry = pgstat_create_my_per_backend_entry(kind);
+ if (entry == NULL)
+ return NULL;
+ }
+
+ if (nowait)
+ {
+ if (!LWLockConditionalAcquire(&entry->lock, LW_EXCLUSIVE))
+ return NULL;
+ }
+ else
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ return entry;
+}
+
+/*
+ * Accumulate all live per-backend entries into the kind's data in
+ * pgStatLocal.snapshot, optionally caching each entry for SNAPSHOT mode.
+ */
+void
+pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash, void *snap)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ dshash_seq_status hstat;
+ PgStatShared_PerBackendEntry *entry;
+
+ dshash_seq_init(&hstat, hash, false);
+
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->lock, LW_SHARED);
+
+ /* Kind-specific accumulation into global snapshot */
+ kind_info->per_backend_acc_cb(snap, entry);
+
+ /*
+ * In SNAPSHOT mode, cache each per-backend entry so that
+ * pgstat_fetch_per_backend() can return a consistent point-in-time
+ * view without re-reading from shared memory.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ {
+ pgstat_cache_per_backend_entry(kind, entry->key,
+ (char *) entry + kind_info->per_backend_data_off);
+ }
+
+ LWLockRelease(&entry->lock);
+ }
+
+ dshash_seq_term(&hstat);
+}
+
+/*
+ * Fetch per-backend stats for the given kind and ProcNumber.
+ * Returns NULL if no entry exists. In NONE mode, returns a copy allocated in
+ * the current memory context. In CACHE and SNAPSHOT modes, returns a pointer
+ * owned by the statistics snapshot cache.
+ */
+void *
+pgstat_fetch_per_backend(PgStat_Kind kind, ProcNumber procnum)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ dshash_table *hash;
+ PgStatShared_PerBackendEntry *entry;
+ void *stats_data;
+ PgStat_PerBackendSnapshotEntry *snapshot_entry;
+
+ pgstat_maybe_clear_snapshot();
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return NULL;
+
+ /* In NONE mode, read directly and don't cache */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
+ {
+ entry = dshash_find(hash, &procnum, false);
+ if (entry == NULL)
+ return NULL;
+
+ LWLockAcquire(&entry->lock, LW_SHARED);
+ stats_data = palloc(kind_info->per_backend_data_len);
+ memcpy(stats_data, (char *) entry + kind_info->per_backend_data_off,
+ kind_info->per_backend_data_len);
+ LWLockRelease(&entry->lock);
+ dshash_release_lock(hash, entry);
+
+ return stats_data;
+ }
+
+ Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE ||
+ pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT);
+
+ /*
+ * Building a full snapshot pre-caches all existing per-backend entries.
+ * CACHE mode only needs the requested entry, so it must not build the
+ * aggregate fixed-kind snapshot and scan every live backend.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ pgstat_snapshot_fixed(kind);
+
+ snapshot_entry = pgstat_lookup_per_backend_entry(kind, procnum);
+
+ if (snapshot_entry != NULL)
+ return snapshot_entry->data;
+
+ /*
+ * Once a full snapshot has been built, a cache miss means the entry did
+ * not exist at the snapshot point. Do not admit an entry created later.
+ */
+ if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT)
+ return NULL;
+
+ /* CACHE miss: copy the live entry directly into the snapshot context. */
+ entry = dshash_find(hash, &procnum, false);
+
+ if (entry == NULL)
+ return pgstat_cache_per_backend_entry(kind, procnum, NULL);
+
+ LWLockAcquire(&entry->lock, LW_SHARED);
+
+ stats_data = pgstat_cache_per_backend_entry(kind, procnum,
+ (char *) entry + kind_info->per_backend_data_off);
+
+ LWLockRelease(&entry->lock);
+ dshash_release_lock(hash, entry);
+
+ return stats_data;
+}
+
+/*
+ * Accumulate this process's per-backend stats into the global stats, then
+ * remove the entry from the dshash.
+ * Acquire the kind lock before the dshash partition lock. Snapshots and resets
+ * hold the kind lock while accessing both the global stats and live entries,
+ * so an entry cannot move between them during either operation.
+ *
+ * NB: The entry may belong to an earlier process that used the same ProcNumber.
+ * It must still be accumulated before removal.
+ */
+void
+pgstat_acc_my_per_backend(PgStat_Kind kind, LWLock *lock)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table *hash;
+ void *dst;
+ PgStatShared_PerBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return;
+
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ dst = shared_struct + kind_info->shared_data_off;
+
+ LWLockAcquire(lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &MyProcNumber, true);
+
+ if (entry == NULL)
+ {
+ state->my_entry = NULL;
+ LWLockRelease(lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ if (state->my_entry == entry)
+ state->my_entry = NULL;
+
+ /* Kind-specific accumulation */
+ kind_info->per_backend_acc_cb(dst, entry);
+
+ LWLockRelease(&entry->lock);
+
+ /* Remove the entry */
+ dshash_delete_entry(hash, entry);
+
+ LWLockRelease(lock);
+}
+
+/*
+ * Accumulate all per-backend entries into global stats and delete them.
+ * Called at clean server shutdown before writing the stats file. Acquire the
+ * kind's global lock before starting the dshash scan.
+ */
+void
+pgstat_acc_all_per_backend(PgStat_Kind kind, LWLock *lock)
+{
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind);
+ char *shared_struct;
+ dshash_table *hash;
+ void *dst;
+ dshash_seq_status hstat;
+ PgStatShared_PerBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(kind);
+
+ if (hash == NULL)
+ return;
+
+ shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off;
+ dst = shared_struct + kind_info->shared_data_off;
+
+ LWLockAcquire(lock, LW_EXCLUSIVE);
+
+ dshash_seq_init(&hstat, hash, true);
+
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->lock, LW_EXCLUSIVE);
+
+ if (state->my_entry == entry)
+ state->my_entry = NULL;
+
+ /* Kind-specific accumulation */
+ kind_info->per_backend_acc_cb(dst, entry);
+
+ LWLockRelease(&entry->lock);
+ dshash_delete_current(&hstat);
+ }
+
+ dshash_seq_term(&hstat);
+
+ LWLockRelease(lock);
+}
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 5feed5f842a..0e7c92f069b 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -168,6 +168,47 @@ StatsShmemRequest(void *arg)
);
}
+/*
+ * Create a dshash for each built-in kind that stores per-backend statistics.
+ * Derive the entry size and handle location from the kind metadata, just as
+ * attachment does.
+ */
+static void
+pgstat_create_per_backend_hashes(dsa_area *dsa, PgStat_ShmemControl *ctl)
+{
+ dshash_parameters params = {
+ sizeof(ProcNumber),
+ 0,
+ dshash_memcmp,
+ dshash_memhash,
+ dshash_memcpy,
+ LWTRANCHE_PGSTATS_HASH
+ };
+
+ for (PgStat_Kind kind = PGSTAT_KIND_BUILTIN_MIN;
+ kind <= PGSTAT_KIND_BUILTIN_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ char *shared_struct;
+ dshash_table_handle *handle_ptr;
+ dshash_table *dsh;
+
+ if (kind_info == NULL || kind_info->per_backend_data_len == 0)
+ continue;
+
+ shared_struct = (char *) ctl + kind_info->shared_ctl_off;
+ handle_ptr = (dshash_table_handle *)
+ (shared_struct + kind_info->per_backend_hash_handle_off);
+
+ params.entry_size = kind_info->per_backend_data_off +
+ kind_info->per_backend_data_len;
+
+ dsh = dshash_create(dsa, ¶ms, NULL);
+ *handle_ptr = dshash_get_hash_table_handle(dsh);
+ dshash_detach(dsh);
+ }
+}
+
/*
* Initialize cumulative statistics system during startup
*/
@@ -210,6 +251,11 @@ StatsShmemInit(void *arg)
/* lift limit set above */
dsa_set_size_limit(dsa, -1);
+ /*
+ * Create per-backend hashes while the local DSA reference is available.
+ */
+ pgstat_create_per_backend_hashes(dsa, ctl);
+
/*
* Postmaster will never access these again, thus free the local
* dsa/dshash references.
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 14369e59a1c..6317f913fdc 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -380,6 +380,26 @@ typedef struct PgStat_KindInfo
*/
void (*snapshot_cb) (void);
+ /*
+ * Per-backend dshash support for built-in fixed-numbered statistics kinds
+ * that also maintain per-backend entries in a dedicated dshash. If
+ * per_backend_data_len is non-zero, the generic infrastructure handles
+ * attach, fetch, accumulate, and snapshot pre-caching automatically.
+ *
+ * Each entry starts with PgStatShared_PerBackendEntry, followed by the
+ * kind-specific statistics payload.
+ */
+ uint32 per_backend_data_off; /* offset of stats data in entry */
+ uint32 per_backend_data_len; /* size of stats data in entry */
+ /* offset of dshash_table_handle in shared struct */
+ uint32 per_backend_hash_handle_off;
+
+ /*
+ * Callback to accumulate one per-backend entry into a destination of the
+ * kind's statistics type. Called with the entry's content lock held.
+ */
+ void (*per_backend_acc_cb) (void *dst, void *entry);
+
/* name of the kind of stats */
const char *const name;
} PgStat_KindInfo;
@@ -479,6 +499,17 @@ typedef struct PgStatShared_SLRU
PgStat_SLRUStats stats[SLRU_NUM_ELEMENTS];
} PgStatShared_SLRU;
+/*
+ * Common header for entries in per-backend statistics dshashes. The
+ * ProcNumber key must be the first field for dshash.
+ */
+typedef struct PgStatShared_PerBackendEntry
+{
+ ProcNumber key;
+ BackendType backend_type;
+ LWLock lock;
+} PgStatShared_PerBackendEntry;
+
typedef struct PgStatShared_Wal
{
/* lock protects ->stats */
@@ -623,6 +654,9 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ /* Per-backend snapshot hash */
+ struct pgstat_per_backend_snapshot_hash *per_backend_stats;
+
/*
* Data in snapshot for custom fixed-numbered statistics, indexed by
* (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
@@ -696,6 +730,17 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid,
bool *may_free);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void pgstat_maybe_clear_snapshot(void);
+
+/* Generic per-backend helpers */
+extern void pgstat_create_my_per_backend_entries(void);
+extern dshash_table *pgstat_per_backend_attach(PgStat_Kind kind);
+extern void *pgstat_lock_my_per_backend_entry(PgStat_Kind kind, bool nowait);
+extern void pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash,
+ void *snap);
+extern void *pgstat_fetch_per_backend(PgStat_Kind kind, ProcNumber procnum);
+extern void pgstat_acc_my_per_backend(PgStat_Kind kind, LWLock *lock);
+extern void pgstat_acc_all_per_backend(PgStat_Kind kind, LWLock *lock);
/*
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 5d432074c2c..66f361e29ef 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2303,6 +2303,7 @@ PgStatShared_HashEntry
PgStatShared_Index
PgStatShared_IO
PgStatShared_Lock
+PgStatShared_PerBackendEntry
PgStatShared_Relation
PgStatShared_ReplSlot
PgStatShared_SLRU
@@ -2328,6 +2329,9 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_Lock
PgStat_LockEntry
+PgStat_PerBackendLocalState
+PgStat_PerBackendSnapshotEntry
+PgStat_PerBackendSnapshotKey
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
--
2.34.1
[text/x-diff] v3-0003-pgstat-move-WAL-statistics-to-new-per-backend-inf.patch (20.8K, ../../arDZoI%2F5xHU2utRh@bdtpg/4-v3-0003-pgstat-move-WAL-statistics-to-new-per-backend-inf.patch)
download | inline diff:
From b6e1868453ba13e47d9f2433f360756acadbe9eb Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Wed, 29 Jul 2026 14:13:28 +0000
Subject: [PATCH v3 3/6] pgstat: move WAL statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's WAL, Lock, and IO statistics
together in one variable-numbered entry keyed by ProcNumber. Move WAL
statistics into a dedicated ProcNumber keyed dshash associated with the
fixed WAL statistics kind. The global WAL stats hold data for backends that
have exited, while the dshash holds statistics for live backends.
Build pg_stat_wal snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove WAL counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_wal(). This also makes WAL statistics available for auxiliary
and shared memory-only workers. Update the documentation to describe the new
behavior.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
doc/src/sgml/monitoring.sgml | 7 +-
src/backend/utils/activity/pgstat.c | 18 ++
src/backend/utils/activity/pgstat_backend.c | 75 --------
src/backend/utils/activity/pgstat_wal.c | 179 +++++++++++++++++---
src/backend/utils/adt/pgstatfuncs.c | 37 +++-
src/include/pgstat.h | 3 +-
src/include/utils/pgstat_internal.h | 22 ++-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 232 insertions(+), 110 deletions(-)
3.6% doc/src/sgml/
78.0% src/backend/utils/activity/
10.2% src/backend/utils/adt/
6.0% src/include/utils/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 62dadf3e86c..a94fcc64fe1 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5849,10 +5849,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns WAL statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_wal</structname> view.
- </para>
- <para>
- The function does not return WAL statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher.
</para></entry>
</row>
@@ -6028,7 +6024,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>wal</literal>: Reset all the counters shown in the
- <structname>pg_stat_wal</structname> view.
+ <structname>pg_stat_wal</structname> view, as well as per-backend
+ WAL statistics returned by <function>pg_stat_get_backend_wal</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 99ac24f166d..2b26d8cf06d 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -516,6 +516,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_wal_init_shmem_cb,
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_WalBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_WalStats),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_Wal, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_wal_per_backend_acc_cb,
},
};
@@ -627,6 +632,9 @@ pgstat_before_server_shutdown(int code, Datum arg)
*/
if (code == 0)
{
+ /* Transfer all live per-backend stats before writing the stats file. */
+ pgstat_wal_acc_all_backends();
+
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
}
@@ -670,6 +678,9 @@ pgstat_shutdown_hook(int code, Datum arg)
if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
pgstat_request_entry_refs_gc();
+ /* Accumulate per-backend WAL stats into the global stats */
+ pgstat_wal_acc_backend_cb();
+
pgstat_detach_shmem();
#ifdef USE_ASSERT_CHECKING
@@ -690,6 +701,13 @@ pgstat_initialize(void)
pgstat_attach_shmem();
+ /*
+ * NB: need to accept that there might be stats from an older backend that
+ * used the same proc number. Accumulate them into the global stats before
+ * we start using the entry.
+ */
+ pgstat_wal_acc_backend_cb();
+
/*
* Create and cache per-backend statistics entries here. This also covers
* processes that never call InitPostgres(), such as shared-memory-only
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index b736b2ccc6f..76b970a8c41 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -41,14 +41,6 @@ static PgStat_BackendPending PendingBackendStats;
static bool backend_has_iostats = false;
static bool backend_has_lockstats = false;
-/*
- * WAL usage counters saved from pgWalUsage at the previous call to
- * pgstat_flush_backend(). This is used to calculate how much WAL usage
- * happens between pgstat_flush_backend() calls, by subtracting the
- * previous counters from the current ones.
- */
-static WalUsage prevBackendWalUsage;
-
/*
* Utility routines to report I/O stats for backends, kept here to avoid
* exposing PendingBackendStats to the outside world.
@@ -244,58 +236,6 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
backend_has_iostats = false;
}
-/*
- * To determine whether WAL usage happened.
- */
-static inline bool
-pgstat_backend_wal_have_pending(void)
-{
- return (pgWalUsage.wal_records != prevBackendWalUsage.wal_records);
-}
-
-/*
- * Flush out locally pending backend WAL statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_WalCounters *bktype_shstats;
- WalUsage wal_usage_diff = {0};
-
- /*
- * This function can be called even if nothing at all has happened for WAL
- * statistics. In this case, avoid unnecessarily modifying the stats
- * entry.
- */
- if (!pgstat_backend_wal_have_pending())
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.wal_counters;
-
- /*
- * Calculate how much WAL usage counters were increased by subtracting the
- * previous counters from the current ones.
- */
- WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevBackendWalUsage);
-
-#define WALSTAT_ACC(fld, var_to_add) \
- (bktype_shstats->fld += var_to_add.fld)
- WALSTAT_ACC(wal_buffers_full, wal_usage_diff);
- WALSTAT_ACC(wal_records, wal_usage_diff);
- WALSTAT_ACC(wal_fpi, wal_usage_diff);
- WALSTAT_ACC(wal_bytes, wal_usage_diff);
- WALSTAT_ACC(wal_fpi_bytes, wal_usage_diff);
-#undef WALSTAT_ACC
-
- /*
- * Save the current counters for the subsequent calculation of WAL usage.
- */
- prevBackendWalUsage = pgWalUsage;
-}
-
/*
* Flush out locally pending backend lock statistics. Locking is managed
* by the caller.
@@ -345,11 +285,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
has_pending_data = true;
- /* Some WAL data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_WAL) &&
- pgstat_backend_wal_have_pending())
- has_pending_data = true;
-
/* Some lock data pending? */
if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats)
has_pending_data = true;
@@ -366,9 +301,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_IO)
pgstat_flush_backend_entry_io(entry_ref);
- if (flags & PGSTAT_BACKEND_FLUSH_WAL)
- pgstat_flush_backend_entry_wal(entry_ref);
-
if (flags & PGSTAT_BACKEND_FLUSH_LOCK)
pgstat_flush_backend_entry_lock(entry_ref);
@@ -411,13 +343,6 @@ pgstat_create_backend(ProcNumber procnum)
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
backend_has_iostats = false;
backend_has_lockstats = false;
-
- /*
- * Initialize prevBackendWalUsage with pgWalUsage so that
- * pgstat_backend_flush_cb() can calculate how much pgWalUsage counters
- * are increased by subtracting prevBackendWalUsage from pgWalUsage.
- */
- prevBackendWalUsage = pgWalUsage;
}
/*
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..ece4d91ed70 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -8,6 +8,11 @@
* storage implementation and the details about individual types of
* statistics.
*
+ * WAL statistics use a per-backend dshash to avoid double-counting. Each
+ * backend flushes WAL usage to its own entry in the dshash (keyed by
+ * ProcNumber). The global pg_stat_wal view aggregates the global stats
+ * (which holds stats from exited backends) plus all live per-backend entries.
+ *
* Copyright (c) 2001-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -52,7 +57,6 @@ pgstat_report_wal(bool force)
/* flush wal stats */
(void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
/* flush IO stats */
pgstat_flush_io(nowait);
@@ -84,13 +88,15 @@ pgstat_wal_have_pending(void)
* Calculate how much WAL usage counters have increased by subtracting the
* previous counters from the current ones.
*
+ * Flush WAL usage counters to the per-backend dshash entry.
+ *
* If nowait is true, this function returns true if the lock could not be
* acquired. Otherwise return false.
*/
bool
pgstat_wal_flush_cb(bool nowait)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_WalBackendEntry *entry;
WalUsage wal_usage_diff = {0};
Assert(IsUnderPostmaster || !IsPostmasterEnvironment);
@@ -105,19 +111,18 @@ pgstat_wal_flush_cb(bool nowait)
return false;
/*
- * We don't update the WAL usage portion of the local WalStats elsewhere.
* Calculate how much WAL usage counters were increased by subtracting the
* previous counters from the current ones.
*/
WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevWalUsage);
- if (!nowait)
- LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
- return true;
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_WAL, nowait);
+
+ if (entry == NULL)
+ return nowait;
#define WALSTAT_ACC(fld, var_to_add) \
- (stats_shmem->stats.wal_counters.fld += var_to_add.fld)
+ (entry->stats.wal_counters.fld += var_to_add.fld)
WALSTAT_ACC(wal_records, wal_usage_diff);
WALSTAT_ACC(wal_fpi, wal_usage_diff);
WALSTAT_ACC(wal_bytes, wal_usage_diff);
@@ -125,7 +130,7 @@ pgstat_wal_flush_cb(bool nowait)
WALSTAT_ACC(wal_buffers_full, wal_usage_diff);
#undef WALSTAT_ACC
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&entry->header.lock);
/*
* Save the current counters for the subsequent calculation of WAL usage.
@@ -157,21 +162,157 @@ pgstat_wal_init_shmem_cb(void *stats)
void
pgstat_wal_reset_all_cb(TimestampTz ts)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ dshash_seq_status hstat;
+ PgStatShared_WalBackendEntry *entry;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
+
+ /* Reset all per-backend entries */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.wal_counters, 0, sizeof(PgStat_WalCounters));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
+ }
- LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
- memset(&stats_shmem->stats, 0, sizeof(stats_shmem->stats));
- stats_shmem->stats.stat_reset_timestamp = ts;
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build WAL stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_wal_snapshot_cb(void)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ PgStat_WalStats *snap = &pgStatLocal.snapshot.wal;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_WalStats));
+
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_WAL, hash, snap);
- LWLockAcquire(&stats_shmem->lock, LW_SHARED);
- memcpy(&pgStatLocal.snapshot.wal, &stats_shmem->stats,
- sizeof(pgStatLocal.snapshot.wal));
- LWLockRelease(&stats_shmem->lock);
+ LWLockRelease(&shmem->lock);
+}
+
+/* Macro to accumulate WAL counters from src into dst */
+#define WAL_ACCUMULATE_COUNTERS(dst, src) \
+do { \
+ (dst).wal_records += (src).wal_records; \
+ (dst).wal_fpi += (src).wal_fpi; \
+ (dst).wal_bytes += (src).wal_bytes; \
+ (dst).wal_fpi_bytes += (src).wal_fpi_bytes; \
+ (dst).wal_buffers_full += (src).wal_buffers_full; \
+} while (0)
+
+/*
+ * Accumulate one per-backend WAL entry into a snapshot or the global stats.
+ */
+void
+pgstat_wal_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_WalStats *stats = dst;
+ PgStatShared_WalBackendEntry *e = (PgStatShared_WalBackendEntry *) entry;
+
+ WAL_ACCUMULATE_COUNTERS(stats->wal_counters, e->stats.wal_counters);
+}
+
+/*
+ * Accumulate a backend's WAL stats into the global stats, then
+ * remove the entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_wal_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_WAL, &pgStatLocal.shmem->wal.lock);
+}
+
+/*
+ * Returns per-backend WAL statistics for the given ProcNumber.
+ */
+PgStat_WalStats *
+pgstat_fetch_stat_backend_wal(ProcNumber procnum)
+{
+ return (PgStat_WalStats *) pgstat_fetch_per_backend(PGSTAT_KIND_WAL, procnum);
+}
+
+/*
+ * Reset a backend's WAL stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_wal_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal;
+ dshash_table *hash;
+ PgStatShared_WalBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ WAL_ACCUMULATE_COUNTERS(shmem->stats.wal_counters, entry->stats.wal_counters);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats.wal_counters, 0, sizeof(PgStat_WalCounters));
+ entry->stats.stat_reset_timestamp = ts;
+
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
+}
+
+/*
+ * Accumulate all per-backend WAL stats entries into the global stats and remove
+ * them. Called at clean server shutdown to ensure all flushed data is preserved
+ * in the stats file.
+ */
+void
+pgstat_wal_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_WAL, &pgStatLocal.shmem->wal.lock);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 64b6f60516c..ef541205db4 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1769,19 +1769,33 @@ Datum
pg_stat_get_backend_wal(PG_FUNCTION_ARGS)
{
int pid;
- PgStat_Backend *backend_stats;
- PgStat_WalCounters bktype_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ PgStat_WalStats *wal_stats;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, NULL);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
+ PG_RETURN_NULL();
+
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
PG_RETURN_NULL();
- bktype_stats = backend_stats->wal_counters;
+ wal_stats = pgstat_fetch_stat_backend_wal(procnum);
- /* save tuples with data from this PgStat_WalCounters */
- return (pg_stat_wal_build_tuple(bktype_stats, backend_stats->stat_reset_timestamp));
+ if (!wal_stats)
+ PG_RETURN_NULL();
+
+ return (pg_stat_wal_build_tuple(wal_stats->wal_counters,
+ wal_stats->stat_reset_timestamp));
}
/*
@@ -2175,6 +2189,7 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
PGPROC *proc;
PgBackendStatus *beentry;
ProcNumber procNumber;
+ TimestampTz ts;
int backend_pid = PG_GETARG_INT32(0);
proc = BackendPidGetProc(backend_pid);
@@ -2196,6 +2211,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
PG_RETURN_VOID();
+ /*
+ * Accumulate the backend's WAL stats into the global stats, then zero the
+ * entry.
+ */
+ ts = GetCurrentTimestamp();
+ pgstat_wal_reset_backend_cb(procNumber, ts);
+
+ /* Reset IO and Lock stats still in PGSTAT_KIND_BACKEND */
pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
PG_RETURN_VOID();
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 187d82c96fe..9a71b8d3cd8 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -593,7 +593,6 @@ typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
PgStat_BktypeIO io_stats;
- PgStat_WalCounters wal_counters;
PgStat_PendingLock lock_stats;
} PgStat_Backend;
@@ -946,6 +945,8 @@ extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_
extern void pgstat_report_wal(bool force);
extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
+extern PgStat_WalStats *pgstat_fetch_stat_backend_wal(ProcNumber procnum);
+extern void pgstat_wal_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
/*
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 6317f913fdc..e4ce6d3ce71 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -510,11 +510,25 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for WAL statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_WalBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_WalStats stats;
+} PgStatShared_WalBackendEntry;
+
typedef struct PgStatShared_Wal
{
- /* lock protects ->stats */
LWLock lock;
PgStat_WalStats stats;
+
+ /*
+ * Per-backend dshash, keyed by ProcNumber.
+ */
+ dshash_table_handle backend_hash_handle;
} PgStatShared_Wal;
@@ -757,9 +771,8 @@ extern void pgstat_archiver_snapshot_cb(void);
/* flags for pgstat_flush_backend() */
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL | PGSTAT_BACKEND_FLUSH_LOCK)
+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_LOCK)
extern bool pgstat_flush_backend(bool nowait, uint32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
@@ -907,6 +920,9 @@ extern bool pgstat_wal_flush_cb(bool nowait);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
+extern void pgstat_wal_acc_backend_cb(void);
+extern void pgstat_wal_acc_all_backends(void);
+extern void pgstat_wal_per_backend_acc_cb(void *dst, void *entry);
/*
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 66f361e29ef..be4f2ad821c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2309,6 +2309,7 @@ PgStatShared_ReplSlot
PgStatShared_SLRU
PgStatShared_Subscription
PgStatShared_Wal
+PgStatShared_WalBackendEntry
PgStat_ArchiverStats
PgStat_Backend
PgStat_BackendPending
--
2.34.1
[text/x-diff] v3-0004-pgstat-move-Lock-statistics-to-new-per-backend-in.patch (21.3K, ../../arDZoI%2F5xHU2utRh@bdtpg/5-v3-0004-pgstat-move-Lock-statistics-to-new-per-backend-in.patch)
download | inline diff:
From d2e023259fed1132205daec3222d0f90c8e9ee64 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:25:10 +0000
Subject: [PATCH v3 4/6] pgstat: move Lock statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's Lock and IO statistics together in one
variable-numbered entry keyed by ProcNumber. Move Lock statistics into a dedicated
ProcNumber keyed dshash associated with the fixed Lock statistics kind. The global
Lock stats hold data for backends that have exited, while the dshash holds
statistics for live backends.
Build pg_stat_lock snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove Lock counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_lock(). This also makes Lock statistics available for
auxiliary and shared memory-only workers. Update the documentation to describe the
new behavior.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/utils/activity/pgstat.c | 10 +-
src/backend/utils/activity/pgstat_backend.c | 72 --------
src/backend/utils/activity/pgstat_lock.c | 183 +++++++++++++++++---
src/backend/utils/adt/pgstatfuncs.c | 34 +++-
src/include/pgstat.h | 12 +-
src/include/utils/pgstat_internal.h | 21 ++-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 218 insertions(+), 123 deletions(-)
3.6% doc/src/sgml/
74.4% src/backend/utils/activity/
9.8% src/backend/utils/adt/
6.9% src/include/utils/
4.7% src/include/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a94fcc64fe1..f0f86e63fe7 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5830,10 +5830,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns lock statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_lock</structname> view.
- </para>
- <para>
- The function does not return lock statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher.
</para></entry>
</row>
@@ -6006,7 +6002,9 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>lock</literal>: Reset all the counters shown in the
- <structname>pg_stat_lock</structname> view.
+ <structname>pg_stat_lock</structname> view, as well as per-backend
+ lock statistics returned by
+ <function>pg_stat_get_backend_lock</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 2b26d8cf06d..6a125f48c56 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -481,6 +481,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_lock_init_shmem_cb,
.reset_all_cb = pgstat_lock_reset_all_cb,
.snapshot_cb = pgstat_lock_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_LockBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_Lock),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_Lock, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_lock_per_backend_acc_cb,
},
[PGSTAT_KIND_SLRU] = {
@@ -634,6 +639,7 @@ pgstat_before_server_shutdown(int code, Datum arg)
{
/* Transfer all live per-backend stats before writing the stats file. */
pgstat_wal_acc_all_backends();
+ pgstat_lock_acc_all_backends();
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
@@ -678,8 +684,9 @@ pgstat_shutdown_hook(int code, Datum arg)
if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
pgstat_request_entry_refs_gc();
- /* Accumulate per-backend WAL stats into the global stats */
+ /* Accumulate per-backend stats into the global stats */
pgstat_wal_acc_backend_cb();
+ pgstat_lock_acc_backend_cb();
pgstat_detach_shmem();
@@ -707,6 +714,7 @@ pgstat_initialize(void)
* we start using the entry.
*/
pgstat_wal_acc_backend_cb();
+ pgstat_lock_acc_backend_cb();
/*
* Create and cache per-backend statistics entries here. This also covers
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 76b970a8c41..8dda2cd88c1 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -39,7 +39,6 @@
*/
static PgStat_BackendPending PendingBackendStats;
static bool backend_has_iostats = false;
-static bool backend_has_lockstats = false;
/*
* Utility routines to report I/O stats for backends, kept here to avoid
@@ -79,39 +78,6 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
pgstat_report_fixed = true;
}
-/*
- * Utility routines to report lock stats for backends, kept here to avoid
- * exposing PendingBackendStats to the outside world.
- */
-void
-pgstat_count_backend_lock_waits(uint8 locktag_type, PgStat_Counter usecs)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(locktag_type <= LOCKTAG_LAST_TYPE);
-
- PendingBackendStats.pending_lock.stats[locktag_type].waits++;
- PendingBackendStats.pending_lock.stats[locktag_type].wait_time += usecs;
-
- backend_has_lockstats = true;
- pgstat_report_fixed = true;
-}
-
-void
-pgstat_count_backend_lock_fastpath_exceeded(uint8 locktag_type)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(locktag_type <= LOCKTAG_LAST_TYPE);
-
- PendingBackendStats.pending_lock.stats[locktag_type].fastpath_exceeded++;
-
- backend_has_lockstats = true;
- pgstat_report_fixed = true;
-}
-
/*
* Returns statistics of a backend by proc number.
*/
@@ -236,36 +202,6 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
backend_has_iostats = false;
}
-/*
- * Flush out locally pending backend lock statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_lock(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_PendingLock *bktype_shstats;
-
- if (!backend_has_lockstats)
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.lock_stats;
-
- for (int i = 0; i <= LOCKTAG_LAST_TYPE; i++)
- {
-#define LOCKSTAT_ACC(fld) \
- (bktype_shstats->stats[i].fld += PendingBackendStats.pending_lock.stats[i].fld)
- LOCKSTAT_ACC(waits);
- LOCKSTAT_ACC(wait_time);
- LOCKSTAT_ACC(fastpath_exceeded);
-#undef LOCKSTAT_ACC
- }
-
- MemSet(&PendingBackendStats.pending_lock, 0, sizeof(PgStat_PendingLock));
- backend_has_lockstats = false;
-}
-
/*
* Flush out locally pending backend statistics
*
@@ -285,10 +221,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
has_pending_data = true;
- /* Some lock data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats)
- has_pending_data = true;
-
if (!has_pending_data)
return false;
@@ -301,9 +233,6 @@ pgstat_flush_backend(bool nowait, uint32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_IO)
pgstat_flush_backend_entry_io(entry_ref);
- if (flags & PGSTAT_BACKEND_FLUSH_LOCK)
- pgstat_flush_backend_entry_lock(entry_ref);
-
pgstat_unlock_entry(entry_ref);
return false;
@@ -342,7 +271,6 @@ pgstat_create_backend(ProcNumber procnum)
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
backend_has_iostats = false;
- backend_has_lockstats = false;
}
/*
diff --git a/src/backend/utils/activity/pgstat_lock.c b/src/backend/utils/activity/pgstat_lock.c
index c20c7599683..5aa380a6243 100644
--- a/src/backend/utils/activity/pgstat_lock.c
+++ b/src/backend/utils/activity/pgstat_lock.c
@@ -8,6 +8,12 @@
* access / storage implementation and the details about individual types
* of statistics.
*
+ * Lock statistics use a per-backend dshash to avoid double-counting. Each
+ * backend flushes lock stats (waits, wait_time, fastpath_exceeded per lock
+ * tag type) to its own entry in the dshash. The global pg_stat_lock view
+ * aggregates the global stats (which holds stats from exited backends) plus
+ * all live per-backend entries.
+ *
* Copyright (c) 2021-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -40,7 +46,7 @@ pgstat_lock_flush(bool nowait)
}
/*
- * Flush out locally pending lock statistics
+ * Flush out locally pending lock statistics to the per-backend dshash entry.
*
* If no stats have been recorded, this function returns false.
*
@@ -50,31 +56,27 @@ pgstat_lock_flush(bool nowait)
bool
pgstat_lock_flush_cb(bool nowait)
{
- LWLock *lckstat_lock;
- PgStatShared_Lock *shstats;
+ PgStatShared_LockBackendEntry *entry;
if (!have_lockstats)
return false;
- shstats = &pgStatLocal.shmem->lock;
- lckstat_lock = &shstats->lock;
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_LOCK, nowait);
- if (!nowait)
- LWLockAcquire(lckstat_lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(lckstat_lock, LW_EXCLUSIVE))
- return true;
+ if (entry == NULL)
+ return nowait;
for (int i = 0; i <= LOCKTAG_LAST_TYPE; i++)
{
#define LOCKSTAT_ACC(fld) \
- (shstats->stats.stats[i].fld += PendingLockStats.stats[i].fld)
+ (entry->stats.stats[i].fld += PendingLockStats.stats[i].fld)
LOCKSTAT_ACC(waits);
LOCKSTAT_ACC(wait_time);
LOCKSTAT_ACC(fastpath_exceeded);
#undef LOCKSTAT_ACC
}
- LWLockRelease(lckstat_lock);
+ LWLockRelease(&entry->header.lock);
memset(&PendingLockStats, 0, sizeof(PendingLockStats));
have_lockstats = false;
@@ -93,28 +95,125 @@ pgstat_lock_init_shmem_cb(void *stats)
void
pgstat_lock_reset_all_cb(TimestampTz ts)
{
- LWLock *lckstat_lock = &pgStatLocal.shmem->lock.lock;
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ dshash_seq_status hstat;
+ PgStatShared_LockBackendEntry *entry;
+ dshash_table *hash;
- LWLockAcquire(lckstat_lock, LW_EXCLUSIVE);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
- pgStatLocal.shmem->lock.stats.stat_reset_timestamp = ts;
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
- memset(pgStatLocal.shmem->lock.stats.stats, 0,
- sizeof(pgStatLocal.shmem->lock.stats.stats));
+ /* Reset all per-backend entries, since they contribute to the global view */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.stats, 0, sizeof(entry->stats.stats));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
+ }
- LWLockRelease(lckstat_lock);
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build lock stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_lock_snapshot_cb(void)
{
- LWLock *lckstat_lock = &pgStatLocal.shmem->lock.lock;
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ PgStat_Lock *snap = &pgStatLocal.snapshot.lock;
+ dshash_table *hash;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
+
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_Lock));
+
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_LOCK, hash, snap);
+
+ LWLockRelease(&shmem->lock);
+}
+
+/*
+ * Accumulate one per-backend lock entry into a snapshot or the global stats.
+ */
+void
+pgstat_lock_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_Lock *stats = dst;
+ PgStatShared_LockBackendEntry *e = (PgStatShared_LockBackendEntry *) entry;
+
+ for (int j = 0; j <= LOCKTAG_LAST_TYPE; j++)
+ {
+ stats->stats[j].waits += e->stats.stats[j].waits;
+ stats->stats[j].wait_time += e->stats.stats[j].wait_time;
+ stats->stats[j].fastpath_exceeded += e->stats.stats[j].fastpath_exceeded;
+ }
+}
+
+/* Macro to accumulate lock counters from src into dst */
+#define LOCK_ACCUMULATE_COUNTERS(dst, src) \
+do { \
+ for (int _i = 0; _i <= LOCKTAG_LAST_TYPE; _i++) \
+ { \
+ (dst)[_i].waits += (src)[_i].waits; \
+ (dst)[_i].wait_time += (src)[_i].wait_time; \
+ (dst)[_i].fastpath_exceeded += (src)[_i].fastpath_exceeded; \
+ } \
+} while (0)
- LWLockAcquire(lckstat_lock, LW_SHARED);
+/*
+ * Accumulate a backend's lock stats into the global stats, then
+ * remove the entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_lock_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_LOCK, &pgStatLocal.shmem->lock.lock);
+}
- pgStatLocal.snapshot.lock = pgStatLocal.shmem->lock.stats;
+/*
+ * Accumulate all per-backend lock stats entries into the global stats and remove
+ * them. Called at clean server shutdown to ensure all flushed data is preserved
+ * in the stats file.
+ */
+void
+pgstat_lock_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_LOCK, &pgStatLocal.shmem->lock.lock);
+}
- LWLockRelease(lckstat_lock);
+/*
+ * Returns per-backend lock statistics for the given ProcNumber.
+ */
+PgStat_Lock *
+pgstat_fetch_stat_backend_lock(ProcNumber procnum)
+{
+ return (PgStat_Lock *) pgstat_fetch_per_backend(PGSTAT_KIND_LOCK, procnum);
}
/*
@@ -131,8 +230,6 @@ pgstat_count_lock_fastpath_exceeded(uint8 locktag_type)
PendingLockStats.stats[locktag_type].fastpath_exceeded++;
have_lockstats = true;
pgstat_report_fixed = true;
-
- pgstat_count_backend_lock_fastpath_exceeded(locktag_type);
}
/*
@@ -149,6 +246,44 @@ pgstat_count_lock_waits(uint8 locktag_type, PgStat_Counter usecs)
PendingLockStats.stats[locktag_type].wait_time += usecs;
have_lockstats = true;
pgstat_report_fixed = true;
+}
+
+/*
+ * Reset a backend's lock stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_lock_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock;
+ dshash_table *hash;
+ PgStatShared_LockBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ LOCK_ACCUMULATE_COUNTERS(shmem->stats.stats, entry->stats.stats);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats.stats, 0, sizeof(entry->stats.stats));
+ entry->stats.stat_reset_timestamp = ts;
- pgstat_count_backend_lock_waits(locktag_type, usecs);
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index ef541205db4..7922519f426 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1872,19 +1872,36 @@ pg_stat_get_backend_lock(PG_FUNCTION_ARGS)
{
int pid;
ReturnSetInfo *rsinfo;
- PgStat_Backend *backend_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ PgStat_Lock *lock_stats;
InitMaterializedSRF(fcinfo, 0);
rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, NULL);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
return (Datum) 0;
- pg_stat_lock_build_tuples(rsinfo, backend_stats->lock_stats.stats,
- backend_stats->stat_reset_timestamp);
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
+ return (Datum) 0;
+
+ lock_stats = pgstat_fetch_stat_backend_lock(procnum);
+
+ if (!lock_stats)
+ return (Datum) 0;
+
+ pg_stat_lock_build_tuples(rsinfo, lock_stats->stats,
+ lock_stats->stat_reset_timestamp);
return (Datum) 0;
}
@@ -2212,13 +2229,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
/*
- * Accumulate the backend's WAL stats into the global stats, then zero the
- * entry.
+ * Accumulate the backend's WAL and lock stats into the global stats, then
+ * zero the entries.
*/
ts = GetCurrentTimestamp();
pgstat_wal_reset_backend_cb(procNumber, ts);
+ pgstat_lock_reset_backend_cb(procNumber, ts);
- /* Reset IO and Lock stats still in PGSTAT_KIND_BACKEND */
+ /* Reset IO stats still in PGSTAT_KIND_BACKEND */
pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
PG_RETURN_VOID();
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 9a71b8d3cd8..960f0001bd6 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -593,7 +593,6 @@ typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
PgStat_BktypeIO io_stats;
- PgStat_PendingLock lock_stats;
} PgStat_Backend;
/* ---------
@@ -606,12 +605,6 @@ typedef struct PgStat_BackendPending
* Backend statistics store the same amount of IO data as PGSTAT_KIND_IO.
*/
PgStat_PendingIO pending_io;
-
- /*
- * Backend statistics store the same amount of lock data as
- * PGSTAT_KIND_LOCK.
- */
- PgStat_PendingLock pending_lock;
} PgStat_BackendPending;
/*
@@ -664,9 +657,6 @@ extern void pgstat_count_backend_io_op(IOObject io_object,
IOOp io_op, uint32 cnt,
uint64 bytes);
-/* used by pgstat_lock.c for lock stats tracked in backends */
-extern void pgstat_count_backend_lock_waits(uint8 locktag_type, PgStat_Counter usecs);
-extern void pgstat_count_backend_lock_fastpath_exceeded(uint8 locktag_type);
extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber);
extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid,
@@ -723,6 +713,8 @@ extern void pgstat_count_lock_fastpath_exceeded(uint8 locktag_type);
extern void pgstat_count_lock_waits(uint8 locktag_type,
PgStat_Counter usecs);
extern PgStat_Lock *pgstat_fetch_stat_lock(void);
+extern PgStat_Lock *pgstat_fetch_stat_backend_lock(ProcNumber procnum);
+extern void pgstat_lock_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
/*
* Functions in pgstat_database.c
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index e4ce6d3ce71..da4e55371a4 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -487,9 +487,12 @@ typedef struct PgStatShared_IO
typedef struct PgStatShared_Lock
{
- /* lock protects ->stats */
+ /* lock protects ->stats (global stats) */
LWLock lock;
PgStat_Lock stats;
+
+ /* Per-backend dshash, keyed by ProcNumber */
+ dshash_table_handle backend_hash_handle;
} PgStatShared_Lock;
typedef struct PgStatShared_SLRU
@@ -510,6 +513,16 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for lock statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_LockBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_Lock stats;
+} PgStatShared_LockBackendEntry;
+
/*
* Per-backend entry for WAL statistics, stored in a dshash keyed by
* ProcNumber.
@@ -771,8 +784,7 @@ extern void pgstat_archiver_snapshot_cb(void);
/* flags for pgstat_flush_backend() */
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_LOCK)
+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO)
extern bool pgstat_flush_backend(bool nowait, uint32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
@@ -838,6 +850,9 @@ extern bool pgstat_lock_flush_cb(bool nowait);
extern void pgstat_lock_init_shmem_cb(void *stats);
extern void pgstat_lock_reset_all_cb(TimestampTz ts);
extern void pgstat_lock_snapshot_cb(void);
+extern void pgstat_lock_acc_backend_cb(void);
+extern void pgstat_lock_acc_all_backends(void);
+extern void pgstat_lock_per_backend_acc_cb(void *dst, void *entry);
/*
* Functions in pgstat_relation.c
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index be4f2ad821c..e0689985bf4 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2303,6 +2303,7 @@ PgStatShared_HashEntry
PgStatShared_Index
PgStatShared_IO
PgStatShared_Lock
+PgStatShared_LockBackendEntry
PgStatShared_PerBackendEntry
PgStatShared_Relation
PgStatShared_ReplSlot
--
2.34.1
[text/x-diff] v3-0005-pgstat-move-IO-statistics-to-new-per-backend-infr.patch (33.4K, ../../arDZoI%2F5xHU2utRh@bdtpg/6-v3-0005-pgstat-move-IO-statistics-to-new-per-backend-infr.patch)
download | inline diff:
From 8d5e25bab3bda9405468a0647cd6cdaa49198e02 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Fri, 17 Jul 2026 13:26:07 +0000
Subject: [PATCH v3 5/6] pgstat: move IO statistics to new per-backend
infrastructure
PGSTAT_KIND_BACKEND stores each backend's IO statistics in one variable-numbered
entry keyed by ProcNumber. Move IO statistics into a dedicated ProcNumber keyed
dshash associated with the fixed IO statistics kind. The global IO stats hold data
for backends that have exited, while the dshash holds statistics for live backends.
Build pg_stat_io snapshots by copying the global stats and adding every live
per-backend entry.
Transfer the current process's entry into the global stats before deleting it
at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown
writes the statistics file.
Remove IO counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure
in pg_stat_get_backend_io(). This also makes IO statistics available for auxiliary
and shared memory-only workers. Update the documentation to describe the
resulting new behavior.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
doc/src/sgml/monitoring.sgml | 9 +-
src/backend/replication/walsender.c | 2 -
src/backend/utils/activity/pgstat.c | 9 +-
src/backend/utils/activity/pgstat_backend.c | 155 +------------
src/backend/utils/activity/pgstat_io.c | 225 ++++++++++++++-----
src/backend/utils/activity/pgstat_relation.c | 2 -
src/backend/utils/activity/pgstat_shmem.c | 9 +-
src/backend/utils/activity/pgstat_wal.c | 1 -
src/backend/utils/adt/pgstatfuncs.c | 47 ++--
src/include/pgstat.h | 34 +--
src/include/utils/pgstat_internal.h | 28 ++-
src/test/regress/expected/stats.out | 23 +-
src/test/regress/sql/stats.sql | 11 +-
src/tools/pgindent/typedefs.list | 3 +-
14 files changed, 268 insertions(+), 290 deletions(-)
69.3% src/backend/utils/activity/
8.5% src/backend/utils/adt/
4.9% src/include/utils/
5.5% src/include/
4.3% src/test/regress/expected/
3.2% src/test/regress/sql/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index f0f86e63fe7..a4441bc6b40 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -5793,12 +5793,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
Returns I/O statistics about the backend with the specified
process ID. The output fields are exactly the same as the ones in the
<structname>pg_stat_io</structname> view.
- </para>
- <para>
- The function does not return I/O statistics for the checkpointer,
- the background writer, the startup process and the autovacuum launcher
- as they are already visible in the <structname>pg_stat_io</structname>
- view and there is only one of each.
</para></entry>
</row>
@@ -5996,7 +5990,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<listitem>
<para>
<literal>io</literal>: Reset all the counters shown in the
- <structname>pg_stat_io</structname> view.
+ <structname>pg_stat_io</structname> view, as well as per-backend
+ I/O statistics returned by <function>pg_stat_get_backend_io</function>.
</para>
</listitem>
<listitem>
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index e9331de3df5..70b6f77e01c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -2083,7 +2083,6 @@ WalSndWaitForWal(XLogRecPtr loc)
WALSENDER_STATS_FLUSH_INTERVAL))
{
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
last_flush = now;
}
@@ -3200,7 +3199,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
WALSENDER_STATS_FLUSH_INTERVAL))
{
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
last_flush = now;
}
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 6a125f48c56..c9b194270ce 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -395,7 +395,6 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.shared_data_off = offsetof(PgStatShared_Backend, stats),
.shared_data_len = sizeof(((PgStatShared_Backend *) 0)->stats),
- .flush_static_cb = pgstat_backend_flush_cb,
.reset_timestamp_cb = pgstat_backend_reset_timestamp_cb,
},
@@ -464,6 +463,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.init_shmem_cb = pgstat_io_init_shmem_cb,
.reset_all_cb = pgstat_io_reset_all_cb,
.snapshot_cb = pgstat_io_snapshot_cb,
+
+ .per_backend_data_off = offsetof(PgStatShared_IOBackendEntry, stats),
+ .per_backend_data_len = sizeof(PgStat_BackendIO),
+ .per_backend_hash_handle_off = offsetof(PgStatShared_IO, backend_hash_handle),
+ .per_backend_acc_cb = pgstat_io_per_backend_acc_cb,
},
[PGSTAT_KIND_LOCK] = {
@@ -640,6 +644,7 @@ pgstat_before_server_shutdown(int code, Datum arg)
/* Transfer all live per-backend stats before writing the stats file. */
pgstat_wal_acc_all_backends();
pgstat_lock_acc_all_backends();
+ pgstat_io_acc_all_backends();
pgStatLocal.shmem->is_shutdown = true;
pgstat_write_statsfile();
@@ -687,6 +692,7 @@ pgstat_shutdown_hook(int code, Datum arg)
/* Accumulate per-backend stats into the global stats */
pgstat_wal_acc_backend_cb();
pgstat_lock_acc_backend_cb();
+ pgstat_io_acc_backend_cb();
pgstat_detach_shmem();
@@ -715,6 +721,7 @@ pgstat_initialize(void)
*/
pgstat_wal_acc_backend_cb();
pgstat_lock_acc_backend_cb();
+ pgstat_io_acc_backend_cb();
/*
* Create and cache per-backend statistics entries here. This also covers
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 8dda2cd88c1..b5996d15d8f 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -11,9 +11,7 @@
* This statistics kind uses a proc number as object ID for the hash table
* of pgstats. Entries are created each time a process is spawned, and are
* dropped when the process exits. These are not written to the pgstats file
- * on disk. Pending statistics are managed without direct interactions with
- * PgStat_EntryRef->pending, relying on PendingBackendStats instead so as it
- * is possible to report data within critical sections.
+ * on disk.
*
* Copyright (c) 2001-2026, PostgreSQL Global Development Group
*
@@ -24,60 +22,10 @@
#include "postgres.h"
-#include "access/xlog.h"
-#include "executor/instrument.h"
-#include "storage/bufmgr.h"
#include "storage/proc.h"
#include "storage/procarray.h"
-#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
-/*
- * Backend statistics counts waiting to be flushed out. These counters may be
- * reported within critical sections so we use static memory in order to avoid
- * memory allocation.
- */
-static PgStat_BackendPending PendingBackendStats;
-static bool backend_has_iostats = false;
-
-/*
- * Utility routines to report I/O stats for backends, kept here to avoid
- * exposing PendingBackendStats to the outside world.
- */
-void
-pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
- IOOp io_op, instr_time io_time)
-{
- Assert(track_io_timing || track_wal_io_timing);
-
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
-
- INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
- io_time);
-
- backend_has_iostats = true;
- pgstat_report_fixed = true;
-}
-
-void
-pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
- IOOp io_op, uint32 cnt, uint64 bytes)
-{
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return;
-
- Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op));
-
- PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
- PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
-
- backend_has_iostats = true;
- pgstat_report_fixed = true;
-}
-
/*
* Returns statistics of a backend by proc number.
*/
@@ -151,104 +99,6 @@ pgstat_fetch_stat_backend_by_pid(int pid, BackendType *bktype)
return backend_stats;
}
-/*
- * Flush out locally pending backend IO statistics. Locking is managed
- * by the caller.
- */
-static void
-pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
-{
- PgStatShared_Backend *shbackendent;
- PgStat_BktypeIO *bktype_shstats;
- PgStat_PendingIO pending_io;
-
- /*
- * This function can be called even if nothing at all has happened for IO
- * statistics. In this case, avoid unnecessarily modifying the stats
- * entry.
- */
- if (!backend_has_iostats)
- return;
-
- shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;
- bktype_shstats = &shbackendent->stats.io_stats;
- pending_io = PendingBackendStats.pending_io;
-
- for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
- {
- for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
- {
- for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
- {
- instr_time time;
-
- bktype_shstats->counts[io_object][io_context][io_op] +=
- pending_io.counts[io_object][io_context][io_op];
- bktype_shstats->bytes[io_object][io_context][io_op] +=
- pending_io.bytes[io_object][io_context][io_op];
- time = pending_io.pending_times[io_object][io_context][io_op];
-
- bktype_shstats->times[io_object][io_context][io_op] +=
- INSTR_TIME_GET_MICROSEC(time);
- }
- }
- }
-
- /*
- * Clear out the statistics buffer, so it can be re-used.
- */
- MemSet(&PendingBackendStats.pending_io, 0, sizeof(PgStat_PendingIO));
-
- backend_has_iostats = false;
-}
-
-/*
- * Flush out locally pending backend statistics
- *
- * "flags" parameter controls which statistics to flush. Returns true
- * if some statistics could not be flushed due to lock contention.
- */
-bool
-pgstat_flush_backend(bool nowait, uint32 flags)
-{
- PgStat_EntryRef *entry_ref;
- bool has_pending_data = false;
-
- if (!pgstat_tracks_backend_bktype(MyBackendType))
- return false;
-
- /* Some IO data pending? */
- if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats)
- has_pending_data = true;
-
- if (!has_pending_data)
- return false;
-
- entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid,
- MyProcNumber, nowait);
- if (!entry_ref)
- return true;
-
- /* Flush requested statistics */
- if (flags & PGSTAT_BACKEND_FLUSH_IO)
- pgstat_flush_backend_entry_io(entry_ref);
-
- pgstat_unlock_entry(entry_ref);
-
- return false;
-}
-
-/*
- * Callback to flush out locally pending backend statistics.
- *
- * If some stats could not be flushed due to lock contention, return true.
- */
-bool
-pgstat_backend_flush_cb(bool nowait)
-{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
-}
-
/*
* Create backend statistics entry for proc number.
*/
@@ -268,9 +118,6 @@ pgstat_create_backend(ProcNumber procnum)
*/
memset(&shstatent->stats, 0, sizeof(shstatent->stats));
pgstat_unlock_entry(entry_ref);
-
- MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
- backend_has_iostats = false;
}
/*
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 8ec1aad5078..5e65bc5cb4b 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -7,6 +7,11 @@
* from pgstat.c to enforce the line between the statistics access / storage
* implementation and the details about individual types of statistics.
*
+ * IO statistics use a per-backend dshash to avoid double-counting. Each
+ * process flushes IO stats to its own entry in the dshash (keyed by
+ * ProcNumber). The global pg_stat_io view aggregates the global stats
+ * (which holds stats from exited processes) plus all live per-backend entries.
+ *
* Copyright (c) 2021-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
@@ -68,9 +73,6 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
PendingIOStats.counts[io_object][io_context][io_op] += cnt;
PendingIOStats.bytes[io_object][io_context][io_op] += bytes;
- /* Add the per-backend counts */
- pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
-
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -143,10 +145,6 @@ pgstat_count_io_op_time(IOObject io_object, IOContext io_context, IOOp io_op,
INSTR_TIME_ADD(PendingIOStats.pending_times[io_object][io_context][io_op],
io_time);
-
- /* Add the per-backend count */
- pgstat_count_backend_io_op_time(io_object, io_context, io_op,
- io_time);
}
pgstat_count_io_op(io_object, io_context, io_op, cnt, bytes);
@@ -170,7 +168,7 @@ pgstat_flush_io(bool nowait)
}
/*
- * Flush out locally pending IO statistics
+ * Flush out locally pending IO statistics to the per-backend dshash entry.
*
* If no stats have been recorded, this function returns false.
*
@@ -180,20 +178,18 @@ pgstat_flush_io(bool nowait)
bool
pgstat_io_flush_cb(bool nowait)
{
- LWLock *bktype_lock;
+ PgStatShared_IOBackendEntry *entry;
PgStat_BktypeIO *bktype_shstats;
if (!have_iostats)
return false;
- bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
- bktype_shstats =
- &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+ entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_IO, nowait);
- if (!nowait)
- LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
- else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE))
- return true;
+ if (entry == NULL)
+ return nowait;
+
+ bktype_shstats = &entry->stats.stats;
for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
{
@@ -217,12 +213,9 @@ pgstat_io_flush_cb(bool nowait)
}
}
- Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType));
-
- LWLockRelease(bktype_lock);
+ LWLockRelease(&entry->header.lock);
memset(&PendingIOStats, 0, sizeof(PendingIOStats));
-
have_iostats = false;
return false;
@@ -271,55 +264,70 @@ pgstat_io_init_shmem_cb(void *stats)
{
PgStatShared_IO *stat_shmem = (PgStatShared_IO *) stats;
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- LWLockInitialize(&stat_shmem->locks[i], LWTRANCHE_PGSTATS_DATA);
+ LWLockInitialize(&stat_shmem->lock, LWTRANCHE_PGSTATS_DATA);
}
void
pgstat_io_reset_all_cb(TimestampTz ts)
{
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- {
- LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
- PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ dshash_seq_status hstat;
+ PgStatShared_IOBackendEntry *entry;
+ dshash_table *hash;
- LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
- /*
- * Use the lock in the first BackendType's PgStat_BktypeIO to protect
- * the reset timestamp as well.
- */
- if (i == 0)
- pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+ /*
+ * Hold the kind lock while resetting both the global stats and live
+ * entries. Transfers hold the same lock, so pre-reset counters cannot be
+ * moved into the global stats after it is reset.
+ */
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+ memset(&shmem->stats, 0, sizeof(shmem->stats));
+ shmem->stats.stat_reset_timestamp = ts;
- memset(bktype_shstats, 0, sizeof(*bktype_shstats));
- LWLockRelease(bktype_lock);
+ /* Reset all per-backend entries */
+ if (hash != NULL)
+ {
+ dshash_seq_init(&hstat, hash, true);
+ while ((entry = dshash_seq_next(&hstat)) != NULL)
+ {
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+ memset(&entry->stats.stats, 0, sizeof(PgStat_BktypeIO));
+ entry->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&entry->header.lock);
+ }
+ dshash_seq_term(&hstat);
}
+
+ LWLockRelease(&shmem->lock);
}
+/*
+ * Build IO stats snapshot by aggregating global stats and all live
+ * per-backend entries.
+ */
void
pgstat_io_snapshot_cb(void)
{
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- {
- LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
- PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
- PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ PgStat_IO *snap = &pgStatLocal.snapshot.io;
+ dshash_table *hash;
- LWLockAcquire(bktype_lock, LW_SHARED);
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
- /*
- * Use the lock in the first BackendType's PgStat_BktypeIO to protect
- * the reset timestamp as well.
- */
- if (i == 0)
- pgStatLocal.snapshot.io.stat_reset_timestamp =
- pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+ /*
+ * Prevent entries from moving to the global stats between copying it and
+ * scanning the per-backend hash.
+ */
+ LWLockAcquire(&shmem->lock, LW_SHARED);
+ memcpy(snap, &shmem->stats, sizeof(PgStat_IO));
- /* using struct assignment due to better type safety */
- *bktype_snap = *bktype_shstats;
- LWLockRelease(bktype_lock);
- }
+ /* Add in all live per-backend entries */
+ if (hash != NULL)
+ pgstat_per_backend_snapshot(PGSTAT_KIND_IO, hash, snap);
+
+ LWLockRelease(&shmem->lock);
}
/*
@@ -581,3 +589,116 @@ pgstat_tracks_io_op(BackendType bktype, IOObject io_object,
return true;
}
+
+/*
+ * Accumulate IO counters from src into dst.
+ */
+static inline void
+pgstat_io_accumulate_counters(PgStat_BktypeIO *dst, const PgStat_BktypeIO *src)
+{
+ for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++)
+ {
+ for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++)
+ {
+ for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++)
+ {
+ dst->counts[io_object][io_context][io_op] +=
+ src->counts[io_object][io_context][io_op];
+ dst->bytes[io_object][io_context][io_op] +=
+ src->bytes[io_object][io_context][io_op];
+ dst->times[io_object][io_context][io_op] +=
+ src->times[io_object][io_context][io_op];
+ }
+ }
+ }
+}
+
+/*
+ * Accumulate one per-backend IO entry into a snapshot or the global stats.
+ */
+void
+pgstat_io_per_backend_acc_cb(void *dst, void *entry)
+{
+ PgStat_IO *stats = dst;
+ PgStatShared_IOBackendEntry *e = (PgStatShared_IOBackendEntry *) entry;
+ BackendType bktype = e->header.backend_type;
+
+ if (bktype == B_INVALID)
+ return;
+
+ pgstat_io_accumulate_counters(&stats->stats[bktype], &e->stats.stats);
+}
+
+/*
+ * Accumulate a backend's IO stats into the global stats, then remove the
+ * entry from the dshash.
+ *
+ * Called at backend exit after the final flush, or when a ProcNumber is
+ * being reused.
+ */
+void
+pgstat_io_acc_backend_cb(void)
+{
+ pgstat_acc_my_per_backend(PGSTAT_KIND_IO, &pgStatLocal.shmem->io.lock);
+}
+
+/*
+ * Accumulate all remaining per-backend IO stats entries into the global stats
+ * and remove them. Called at clean server shutdown to ensure all flushed data
+ * is preserved in the stats file.
+ */
+void
+pgstat_io_acc_all_backends(void)
+{
+ pgstat_acc_all_per_backend(PGSTAT_KIND_IO, &pgStatLocal.shmem->io.lock);
+}
+
+/*
+ * Returns per-backend IO statistics for the given ProcNumber.
+ */
+PgStat_BackendIO *
+pgstat_fetch_stat_backend_io(ProcNumber procnum)
+{
+ return (PgStat_BackendIO *) pgstat_fetch_per_backend(PGSTAT_KIND_IO, procnum);
+}
+
+/*
+ * Reset a backend's IO stats. Accumulate the entry's counters into the
+ * global stats, then zero the stats and set the reset timestamp.
+ */
+void
+pgstat_io_reset_backend_cb(ProcNumber procnum, TimestampTz ts)
+{
+ PgStatShared_IO *shmem = &pgStatLocal.shmem->io;
+ dshash_table *hash;
+ PgStatShared_IOBackendEntry *entry;
+
+ hash = pgstat_per_backend_attach(PGSTAT_KIND_IO);
+
+ if (hash == NULL)
+ return;
+
+ LWLockAcquire(&shmem->lock, LW_EXCLUSIVE);
+
+ entry = dshash_find(hash, &procnum, true);
+
+ if (entry == NULL)
+ {
+ LWLockRelease(&shmem->lock);
+ return;
+ }
+
+ LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE);
+
+ /* Accumulate current stats into global before zeroing */
+ pgstat_io_accumulate_counters(&shmem->stats.stats[entry->header.backend_type],
+ &entry->stats.stats);
+
+ /* Zero stats and set reset timestamp */
+ memset(&entry->stats, 0, sizeof(entry->stats));
+ entry->stats.stat_reset_timestamp = ts;
+
+ LWLockRelease(&entry->header.lock);
+ dshash_release_lock(hash, entry);
+ LWLockRelease(&shmem->lock);
+}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 5c70543ba88..4c3d54c350a 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -321,7 +321,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* VACUUM command has processed all tables and committed.
*/
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -417,7 +416,6 @@ pgstat_report_analyze(Relation rel,
/* see pgstat_report_vacuum() */
pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 0e7c92f069b..3c117d29219 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -118,12 +118,11 @@ pgstat_dsa_init_size(void)
/*
* The dshash header / initial buckets array needs to fit into "plain"
* shared memory, but it's beneficial to not need dsm segments
- * immediately. A size of 256kB seems works well and is not
- * disproportional compared to other constant sized shared memory
- * allocations. NB: To avoid DSMs further, the user can configure
- * min_dynamic_shared_memory.
+ * immediately. A size of 1MB works well and is not disproportional
+ * compared to other constant sized shared memory allocations. NB: To
+ * avoid DSMs further, the user can configure min_dynamic_shared_memory.
*/
- sz = 256 * 1024;
+ sz = 1024 * 1024;
Assert(dsa_minimum_size() <= sz);
return MAXALIGN(sz);
}
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index ece4d91ed70..99fd20a04c2 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -60,7 +60,6 @@ pgstat_report_wal(bool force)
/* flush IO stats */
pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 7922519f426..a605454a360 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1671,32 +1671,47 @@ Datum
pg_stat_get_backend_io(PG_FUNCTION_ARGS)
{
ReturnSetInfo *rsinfo;
- BackendType bktype;
int pid;
- PgStat_Backend *backend_stats;
- PgStat_BktypeIO *bktype_stats;
+ PGPROC *proc;
+ ProcNumber procnum;
+ PgBackendStatus *beentry;
+ BackendType bktype;
+ PgStat_BackendIO *backend_io;
InitMaterializedSRF(fcinfo, 0);
rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
pid = PG_GETARG_INT32(0);
- backend_stats = pgstat_fetch_stat_backend_by_pid(pid, &bktype);
- if (!backend_stats)
+ proc = BackendPidGetProc(pid);
+
+ if (!proc)
+ proc = AuxiliaryPidGetProc(pid);
+ if (!proc)
+ return (Datum) 0;
+
+ procnum = GetNumberFromPGProc(proc);
+ beentry = pgstat_get_beentry_by_proc_number(procnum);
+
+ if (!beentry || beentry->st_procpid != pid)
return (Datum) 0;
- bktype_stats = &backend_stats->io_stats;
+ bktype = beentry->st_backendType;
+ backend_io = pgstat_fetch_stat_backend_io(procnum);
+
+ if (!backend_io)
+ return (Datum) 0;
/*
* In Assert builds, we can afford an extra loop through all of the
* counters (in pg_stat_io_build_tuples()), checking that only expected
* stats are non-zero, since it keeps the non-Assert code cleaner.
*/
- Assert(pgstat_bktype_io_stats_valid(bktype_stats, bktype));
+ Assert(pgstat_bktype_io_stats_valid(&backend_io->stats, bktype));
+
+ pg_stat_io_build_tuples(rsinfo, &backend_io->stats, bktype,
+ backend_io->stat_reset_timestamp);
- /* save tuples with data from this PgStat_BktypeIO */
- pg_stat_io_build_tuples(rsinfo, bktype_stats, bktype,
- backend_stats->stat_reset_timestamp);
return (Datum) 0;
}
@@ -2224,20 +2239,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS)
if (!beentry)
PG_RETURN_VOID();
- /* Check if the backend type tracks statistics */
- if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
- PG_RETURN_VOID();
-
/*
- * Accumulate the backend's WAL and lock stats into the global stats, then
- * zero the entries.
+ * Accumulate the backend's WAL, lock and IO stats into the global stats,
+ * then zero the entries.
*/
ts = GetCurrentTimestamp();
pgstat_wal_reset_backend_cb(procNumber, ts);
pgstat_lock_reset_backend_cb(procNumber, ts);
-
- /* Reset IO stats still in PGSTAT_KIND_BACKEND */
- pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber);
+ pgstat_io_reset_backend_cb(procNumber, ts);
PG_RETURN_VOID();
}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 960f0001bd6..86592040b64 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -275,7 +275,7 @@ typedef struct PgStat_TableXactStatus
* ------------------------------------------------------------
*/
-#define PGSTAT_FILE_FORMAT_ID 0x01A5BCBD
+#define PGSTAT_FILE_FORMAT_ID 0x01A5BCBE
typedef struct PgStat_ArchiverStats
{
@@ -390,6 +390,12 @@ typedef struct PgStat_BktypeIO
PgStat_Counter times[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
} PgStat_BktypeIO;
+typedef struct PgStat_BackendIO
+{
+ TimestampTz stat_reset_timestamp;
+ PgStat_BktypeIO stats;
+} PgStat_BackendIO;
+
typedef struct PgStat_PendingIO
{
uint64 bytes[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES];
@@ -592,21 +598,8 @@ typedef struct PgStat_WalStats
typedef struct PgStat_Backend
{
TimestampTz stat_reset_timestamp;
- PgStat_BktypeIO io_stats;
} PgStat_Backend;
-/* ---------
- * PgStat_BackendPending Non-flushed backend stats.
- * ---------
- */
-typedef struct PgStat_BackendPending
-{
- /*
- * Backend statistics store the same amount of IO data as PGSTAT_KIND_IO.
- */
- PgStat_PendingIO pending_io;
-} PgStat_BackendPending;
-
/*
* Functions in pgstat.c
*/
@@ -647,17 +640,6 @@ extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void);
* Functions in pgstat_backend.c
*/
-/* used by pgstat_io.c for I/O stats tracked in backends */
-extern void pgstat_count_backend_io_op_time(IOObject io_object,
- IOContext io_context,
- IOOp io_op,
- instr_time io_time);
-extern void pgstat_count_backend_io_op(IOObject io_object,
- IOContext io_context,
- IOOp io_op, uint32 cnt,
- uint64 bytes);
-
-
extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber);
extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid,
BackendType *bktype);
@@ -694,6 +676,8 @@ extern void pgstat_count_io_op_time(IOObject io_object, IOContext io_context,
uint32 cnt, uint64 bytes);
extern PgStat_IO *pgstat_fetch_stat_io(void);
+extern PgStat_BackendIO *pgstat_fetch_stat_backend_io(ProcNumber procnum);
+extern void pgstat_io_reset_backend_cb(ProcNumber procnum, TimestampTz ts);
extern const char *pgstat_get_io_context_name(IOContext io_context);
extern const char *pgstat_get_io_object_name(IOObject io_object);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index da4e55371a4..54a1746db9d 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -477,12 +477,13 @@ typedef struct PgStatShared_Checkpointer
/* Shared-memory ready PgStat_IO */
typedef struct PgStatShared_IO
{
+ LWLock lock;
+ PgStat_IO stats;
+
/*
- * locks[i] protects stats.stats[i]. locks[0] also protects
- * stats.stat_reset_timestamp.
+ * Per-backend dshash, keyed by ProcNumber.
*/
- LWLock locks[BACKEND_NUM_TYPES];
- PgStat_IO stats;
+ dshash_table_handle backend_hash_handle;
} PgStatShared_IO;
typedef struct PgStatShared_Lock
@@ -513,6 +514,16 @@ typedef struct PgStatShared_PerBackendEntry
LWLock lock;
} PgStatShared_PerBackendEntry;
+/*
+ * Per-backend entry for IO statistics, stored in a dshash keyed by
+ * ProcNumber.
+ */
+typedef struct PgStatShared_IOBackendEntry
+{
+ PgStatShared_PerBackendEntry header;
+ PgStat_BackendIO stats;
+} PgStatShared_IOBackendEntry;
+
/*
* Per-backend entry for lock statistics, stored in a dshash keyed by
* ProcNumber.
@@ -782,12 +793,6 @@ extern void pgstat_archiver_snapshot_cb(void);
* Functions in pgstat_backend.c
*/
-/* flags for pgstat_flush_backend() */
-#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO)
-
-extern bool pgstat_flush_backend(bool nowait, uint32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -841,6 +846,9 @@ extern bool pgstat_io_flush_cb(bool nowait);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
+extern void pgstat_io_acc_backend_cb(void);
+extern void pgstat_io_acc_all_backends(void);
+extern void pgstat_io_per_backend_acc_cb(void *dst, void *entry);
/*
* Functions in pgstat_lock.c
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index e3dd0ce89b0..b4f74f49bcd 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -1956,13 +1956,20 @@ SELECT :io_stats_post_reset < :io_stats_pre_reset;
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset
FROM pg_stat_get_backend_io(pg_backend_pid()) \gset
--- pg_stat_reset_shared() did not reset backend IO stats
-SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset;
+-- pg_stat_reset_shared() also resets per-backend IO stats
+SELECT :my_io_stats_pre_reset > :my_io_stats_post_reset;
?column?
----------
t
(1 row)
+SELECT bool_and(stats_reset IS NOT NULL) AS backend_io_reset_timestamp_set
+ FROM pg_stat_get_backend_io(pg_backend_pid());
+ backend_io_reset_timestamp_set
+--------------------------------
+ t
+(1 row)
+
-- but pg_stat_reset_backend_stats() does
SELECT pg_stat_reset_backend_stats(pg_backend_pid());
pg_stat_reset_backend_stats
@@ -1989,11 +1996,13 @@ SELECT pg_stat_get_backend_io(0);
------------------------
(0 rows)
--- Auxiliary processes return no data.
-SELECT pg_stat_get_backend_io(:checkpointer_pid);
- pg_stat_get_backend_io
-------------------------
-(0 rows)
+-- Auxiliary processes now return data
+SELECT count(*) > 0 AS checkpointer_has_io_stats
+ FROM pg_stat_get_backend_io(:checkpointer_pid);
+ checkpointer_has_io_stats
+---------------------------
+ t
+(1 row)
-- test BRIN index doesn't block HOT update
CREATE TABLE brin_hot (
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql
index f4931fd84de..de8d0fdb04e 100644
--- a/src/test/regress/sql/stats.sql
+++ b/src/test/regress/sql/stats.sql
@@ -924,8 +924,10 @@ SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) +
SELECT :io_stats_post_reset < :io_stats_pre_reset;
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset
FROM pg_stat_get_backend_io(pg_backend_pid()) \gset
--- pg_stat_reset_shared() did not reset backend IO stats
-SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset;
+-- pg_stat_reset_shared() also resets per-backend IO stats
+SELECT :my_io_stats_pre_reset > :my_io_stats_post_reset;
+SELECT bool_and(stats_reset IS NOT NULL) AS backend_io_reset_timestamp_set
+ FROM pg_stat_get_backend_io(pg_backend_pid());
-- but pg_stat_reset_backend_stats() does
SELECT pg_stat_reset_backend_stats(pg_backend_pid());
SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_backend_reset
@@ -935,8 +937,9 @@ SELECT :my_io_stats_pre_reset > :my_io_stats_post_backend_reset;
-- Check invalid input for pg_stat_get_backend_io()
SELECT pg_stat_get_backend_io(NULL);
SELECT pg_stat_get_backend_io(0);
--- Auxiliary processes return no data.
-SELECT pg_stat_get_backend_io(:checkpointer_pid);
+-- Auxiliary processes now return data
+SELECT count(*) > 0 AS checkpointer_has_io_stats
+ FROM pg_stat_get_backend_io(:checkpointer_pid);
-- test BRIN index doesn't block HOT update
CREATE TABLE brin_hot (
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e0689985bf4..111486043fc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2302,6 +2302,7 @@ PgStatShared_Function
PgStatShared_HashEntry
PgStatShared_Index
PgStatShared_IO
+PgStatShared_IOBackendEntry
PgStatShared_Lock
PgStatShared_LockBackendEntry
PgStatShared_PerBackendEntry
@@ -2313,7 +2314,7 @@ PgStatShared_Wal
PgStatShared_WalBackendEntry
PgStat_ArchiverStats
PgStat_Backend
-PgStat_BackendPending
+PgStat_BackendIO
PgStat_BackendSubEntry
PgStat_BgWriterStats
PgStat_BktypeIO
--
2.34.1
[text/x-diff] v3-0006-pgstat-remove-PGSTAT_KIND_BACKEND.patch (13.9K, ../../arDZoI%2F5xHU2utRh@bdtpg/7-v3-0006-pgstat-remove-PGSTAT_KIND_BACKEND.patch)
download | inline diff:
From 27d5e4ee9203a50c9343f490987ae72ed86827ca Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bdrouvot@amazon.com>
Date: Thu, 13 Aug 2026 12:28:55 +0000
Subject: [PATCH v3 6/6] pgstat: remove PGSTAT_KIND_BACKEND
After WAL, Lock, and IO have moved, PGSTAT_KIND_BACKEND contains no data.
Remove that kind and pgstat_backend.c together with their build entries,
shared structures, callbacks, and call sites.
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/anCjoFkGah/uB7lw%40bdtpg
---
src/backend/utils/activity/Makefile | 1 -
src/backend/utils/activity/backend_status.c | 4 -
src/backend/utils/activity/meson.build | 1 -
src/backend/utils/activity/pgstat.c | 20 ---
src/backend/utils/activity/pgstat_backend.c | 179 --------------------
src/include/pgstat.h | 19 ---
src/include/utils/pgstat_internal.h | 13 --
src/include/utils/pgstat_kind.h | 15 +-
src/test/regress/expected/stats.out | 17 +-
src/tools/pgindent/typedefs.list | 2 -
10 files changed, 15 insertions(+), 256 deletions(-)
72.8% src/backend/utils/activity/
9.8% src/include/utils/
6.4% src/include/
10.3% src/test/regress/expected/
diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile
index 8924b10ed2e..618a04602c8 100644
--- a/src/backend/utils/activity/Makefile
+++ b/src/backend/utils/activity/Makefile
@@ -20,7 +20,6 @@ OBJS = \
backend_status.o \
pgstat.o \
pgstat_archiver.o \
- pgstat_backend.o \
pgstat_bgwriter.o \
pgstat_checkpointer.o \
pgstat_database.o \
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index d685fc5cd87..7088fc09651 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -458,10 +458,6 @@ pgstat_bestart_final(void)
PGSTAT_END_WRITE_ACTIVITY(beentry);
- /* Create the backend statistics entry */
- if (pgstat_tracks_backend_bktype(MyBackendType))
- pgstat_create_backend(MyProcNumber);
-
/* Update app name to current GUC setting */
if (application_name)
pgstat_report_appname(application_name);
diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build
index 6434c377e53..ea6495b95d3 100644
--- a/src/backend/utils/activity/meson.build
+++ b/src/backend/utils/activity/meson.build
@@ -5,7 +5,6 @@ backend_sources += files(
'backend_status.c',
'pgstat.c',
'pgstat_archiver.c',
- 'pgstat_backend.c',
'pgstat_bgwriter.c',
'pgstat_checkpointer.c',
'pgstat_database.c',
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index c9b194270ce..132270a3e0e 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -78,7 +78,6 @@
*
* Each statistics kind is handled in a dedicated file:
* - pgstat_archiver.c
- * - pgstat_backend.c
* - pgstat_bgwriter.c
* - pgstat_checkpointer.c
* - pgstat_database.c
@@ -383,21 +382,6 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.reset_timestamp_cb = pgstat_subscription_reset_timestamp_cb,
},
- [PGSTAT_KIND_BACKEND] = {
- .name = "backend",
-
- .fixed_amount = false,
- .write_to_file = false,
-
- .accessed_across_databases = true,
-
- .shared_size = sizeof(PgStatShared_Backend),
- .shared_data_off = offsetof(PgStatShared_Backend, stats),
- .shared_data_len = sizeof(((PgStatShared_Backend *) 0)->stats),
-
- .reset_timestamp_cb = pgstat_backend_reset_timestamp_cb,
- },
-
/* stats for fixed-numbered (mostly 1) objects */
[PGSTAT_KIND_ARCHIVER] = {
@@ -685,10 +669,6 @@ pgstat_shutdown_hook(int code, Datum arg)
Assert(dlist_is_empty(&pgStatPending));
dlist_init(&pgStatPending);
- /* drop the backend stats entry */
- if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false))
- pgstat_request_entry_refs_gc();
-
/* Accumulate per-backend stats into the global stats */
pgstat_wal_acc_backend_cb();
pgstat_lock_acc_backend_cb();
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
deleted file mode 100644
index b5996d15d8f..00000000000
--- a/src/backend/utils/activity/pgstat_backend.c
+++ /dev/null
@@ -1,179 +0,0 @@
-/* -------------------------------------------------------------------------
- *
- * pgstat_backend.c
- * Implementation of backend statistics.
- *
- * This file contains the implementation of backend statistics. It is kept
- * separate from pgstat.c to enforce the line between the statistics access /
- * storage implementation and the details about individual types of
- * statistics.
- *
- * This statistics kind uses a proc number as object ID for the hash table
- * of pgstats. Entries are created each time a process is spawned, and are
- * dropped when the process exits. These are not written to the pgstats file
- * on disk.
- *
- * Copyright (c) 2001-2026, PostgreSQL Global Development Group
- *
- * IDENTIFICATION
- * src/backend/utils/activity/pgstat_backend.c
- * -------------------------------------------------------------------------
- */
-
-#include "postgres.h"
-
-#include "storage/proc.h"
-#include "storage/procarray.h"
-#include "utils/pgstat_internal.h"
-
-/*
- * Returns statistics of a backend by proc number.
- */
-PgStat_Backend *
-pgstat_fetch_stat_backend(ProcNumber procNumber)
-{
- PgStat_Backend *backend_entry;
-
- backend_entry = (PgStat_Backend *) pgstat_fetch_entry(PGSTAT_KIND_BACKEND,
- InvalidOid, procNumber,
- NULL);
-
- return backend_entry;
-}
-
-/*
- * Returns statistics of a backend by pid.
- *
- * This routine includes sanity checks to ensure that the backend exists and
- * is running. "bktype" can be optionally defined to return the BackendType
- * of the backend whose statistics are returned.
- */
-PgStat_Backend *
-pgstat_fetch_stat_backend_by_pid(int pid, BackendType *bktype)
-{
- PGPROC *proc;
- PgBackendStatus *beentry;
- ProcNumber procNumber;
- PgStat_Backend *backend_stats;
-
- proc = BackendPidGetProc(pid);
- if (bktype)
- *bktype = B_INVALID;
-
- /* this could be an auxiliary process */
- if (!proc)
- proc = AuxiliaryPidGetProc(pid);
-
- if (!proc)
- return NULL;
-
- procNumber = GetNumberFromPGProc(proc);
-
- beentry = pgstat_get_beentry_by_proc_number(procNumber);
- if (!beentry)
- return NULL;
-
- /* check if the backend type tracks statistics */
- if (!pgstat_tracks_backend_bktype(beentry->st_backendType))
- return NULL;
-
- /* if PID does not match, leave */
- if (beentry->st_procpid != pid)
- return NULL;
-
- if (bktype)
- *bktype = beentry->st_backendType;
-
- /*
- * Retrieve the entry. Note that "beentry" may be freed depending on the
- * value of stats_fetch_consistency, so do not access it from this point.
- */
- backend_stats = pgstat_fetch_stat_backend(procNumber);
- if (!backend_stats)
- {
- if (bktype)
- *bktype = B_INVALID;
- return NULL;
- }
-
- return backend_stats;
-}
-
-/*
- * Create backend statistics entry for proc number.
- */
-void
-pgstat_create_backend(ProcNumber procnum)
-{
- PgStat_EntryRef *entry_ref;
- PgStatShared_Backend *shstatent;
-
- entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid,
- procnum, false);
- shstatent = (PgStatShared_Backend *) entry_ref->shared_stats;
-
- /*
- * NB: need to accept that there might be stats from an older backend,
- * e.g. if we previously used this proc number.
- */
- memset(&shstatent->stats, 0, sizeof(shstatent->stats));
- pgstat_unlock_entry(entry_ref);
-}
-
-/*
- * Backend statistics are not collected for all BackendTypes.
- *
- * The following BackendTypes do not participate in the backend stats
- * subsystem:
- * - The same and for the same reasons as in pgstat_tracks_io_bktype().
- * - B_BG_WRITER, B_CHECKPOINTER, B_STARTUP and B_AUTOVAC_LAUNCHER because their
- * I/O stats are already visible in pg_stat_io and there is only one of those.
- *
- * Function returns true if BackendType participates in the backend stats
- * subsystem and false if it does not.
- *
- * When adding a new BackendType, also consider adding relevant restrictions to
- * pgstat_tracks_io_object() and pgstat_tracks_io_op().
- */
-bool
-pgstat_tracks_backend_bktype(BackendType bktype)
-{
- /*
- * List every type so that new backend types trigger a warning about
- * needing to adjust this switch.
- */
- switch (bktype)
- {
- case B_INVALID:
- case B_AUTOVAC_LAUNCHER:
- case B_DEAD_END_BACKEND:
- case B_ARCHIVER:
- case B_LOGGER:
- case B_BG_WRITER:
- case B_CHECKPOINTER:
- case B_IO_WORKER:
- case B_STARTUP:
- case B_DATACHECKSUMSWORKER_LAUNCHER:
- case B_DATACHECKSUMSWORKER_WORKER:
- return false;
-
- case B_AUTOVAC_WORKER:
- case B_BACKEND:
- case B_BG_WORKER:
- case B_STANDALONE_BACKEND:
- case B_SLOTSYNC_WORKER:
- case B_WAL_RECEIVER:
- case B_WAL_SENDER:
- case B_WAL_SUMMARIZER:
- case B_WAL_WRITER:
- return true;
- }
-
- return false;
-}
-
-void
-pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts)
-{
- ((PgStatShared_Backend *) header)->stats.stat_reset_timestamp = ts;
-}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 86592040b64..04bb1bbc850 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -591,15 +591,6 @@ typedef struct PgStat_WalStats
TimestampTz stat_reset_timestamp;
} PgStat_WalStats;
-/* -------
- * PgStat_Backend Backend statistics
- * -------
- */
-typedef struct PgStat_Backend
-{
- TimestampTz stat_reset_timestamp;
-} PgStat_Backend;
-
/*
* Functions in pgstat.c
*/
@@ -636,16 +627,6 @@ extern bool pgstat_have_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_report_archiver(const char *xlog, bool failed);
extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void);
-/*
- * Functions in pgstat_backend.c
- */
-
-extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber);
-extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid,
- BackendType *bktype);
-extern bool pgstat_tracks_backend_bktype(BackendType bktype);
-extern void pgstat_create_backend(ProcNumber procnum);
-
/*
* Functions in pgstat_bgwriter.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 54a1746db9d..7fdee3f8337 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -601,12 +601,6 @@ typedef struct PgStatShared_ReplSlot
PgStat_StatReplSlotEntry stats;
} PgStatShared_ReplSlot;
-typedef struct PgStatShared_Backend
-{
- PgStatShared_Common header;
- PgStat_Backend stats;
-} PgStatShared_Backend;
-
/*
* Central shared memory entry for the cumulative stats system.
*
@@ -789,13 +783,6 @@ extern void pgstat_archiver_init_shmem_cb(void *stats);
extern void pgstat_archiver_reset_all_cb(TimestampTz ts);
extern void pgstat_archiver_snapshot_cb(void);
-/*
- * Functions in pgstat_backend.c
- */
-
-extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
- TimestampTz ts);
-
/*
* Functions in pgstat_bgwriter.c
*/
diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h
index 45ca599d0dd..489e8fa38bf 100644
--- a/src/include/utils/pgstat_kind.h
+++ b/src/include/utils/pgstat_kind.h
@@ -30,16 +30,15 @@
#define PGSTAT_KIND_FUNCTION 4 /* per-function statistics */
#define PGSTAT_KIND_REPLSLOT 5 /* per-slot statistics */
#define PGSTAT_KIND_SUBSCRIPTION 6 /* per-subscription statistics */
-#define PGSTAT_KIND_BACKEND 7 /* per-backend statistics */
/* stats for fixed-numbered objects */
-#define PGSTAT_KIND_ARCHIVER 8
-#define PGSTAT_KIND_BGWRITER 9
-#define PGSTAT_KIND_CHECKPOINTER 10
-#define PGSTAT_KIND_IO 11
-#define PGSTAT_KIND_LOCK 12
-#define PGSTAT_KIND_SLRU 13
-#define PGSTAT_KIND_WAL 14
+#define PGSTAT_KIND_ARCHIVER 7
+#define PGSTAT_KIND_BGWRITER 8
+#define PGSTAT_KIND_CHECKPOINTER 9
+#define PGSTAT_KIND_IO 10
+#define PGSTAT_KIND_LOCK 11
+#define PGSTAT_KIND_SLRU 12
+#define PGSTAT_KIND_WAL 13
#define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index b4f74f49bcd..979eca8d6e4 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -120,15 +120,14 @@ SELECT id, name, fixed_amount,
4 | function | f | f | t
5 | replslot | f | t | t
6 | subscription | f | t | t
- 7 | backend | f | t | f
- 8 | archiver | t | f | t
- 9 | bgwriter | t | f | t
- 10 | checkpointer | t | f | t
- 11 | io | t | f | t
- 12 | lock | t | f | t
- 13 | slru | t | f | t
- 14 | wal | t | f | t
-(14 rows)
+ 7 | archiver | t | f | t
+ 8 | bgwriter | t | f | t
+ 9 | checkpointer | t | f | t
+ 10 | io | t | f | t
+ 11 | lock | t | f | t
+ 12 | slru | t | f | t
+ 13 | wal | t | f | t
+(13 rows)
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 111486043fc..b68b823fecd 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2291,7 +2291,6 @@ PgFdwSamplingMethod
PgFdwScanState
PgIfAddrCallback
PgStatShared_Archiver
-PgStatShared_Backend
PgStatShared_BgWriter
PgStatShared_Checkpointer
PgStatShared_Common
@@ -2313,7 +2312,6 @@ PgStatShared_Subscription
PgStatShared_Wal
PgStatShared_WalBackendEntry
PgStat_ArchiverStats
-PgStat_Backend
PgStat_BackendIO
PgStat_BackendSubEntry
PgStat_BgWriterStats
--
2.34.1
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-10 14:45 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-13 15:41 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 07:15 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-09-21 09:17 ` Andres Freund <andres@anarazel.de>
2026-09-22 00:16 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-09-22 03:07 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 2 replies; 11+ messages in thread
From: Andres Freund @ 2026-09-21 09:17 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Sami Imseih <samimseih@gmail.com>
Hi,
On 2026-09-21 07:15:44 +0000, Bertrand Drouvot wrote:
> Subject: [PATCH v3 1/6] pgstat: add tests for per-backend statistics
Good idea to start with that.
> Add an isolation test for per-backend WAL statistics. Verify that SNAPSHOT, CACHE
> and NONE modes behave as expected.
This isn't really related to this patch, but I do wonder if we need to make
this somehow a bit more scalable for testing. If every stats thing has to test
all of these "manually", we have a test-scalability issue...
> Extend test_shm_mq with a shared-memory-only worker that reports WAL usage
> through a routine nonblocking flush and remains alive while the controlling
> backend verifies that the global WAL counters include it.
I'm not entirely sure what this really tests? Couldn't you just test this
e.g. for checkpointer, in a quarter of the lines at most?
> +++ b/src/test/isolation/specs/stats-per-backend.spec
> @@ -0,0 +1,123 @@
> +# Test snapshot, cache, and reset behavior for per-backend statistics.
> +#
> +# Session s2 generates and flushes WAL records, which provide a deterministic
> +# per-backend counter. Session s1 verifies that SNAPSHOT mode fixes all
> +# per-backend entries at the initial statistics snapshot, while CACHE mode
> +# fixes each entry on first access and refreshes it after the transaction
> +# ends. The final permutations verify reset timestamps after per-backend and
> +# shared resets.
> +
> +setup
> +{
> + CREATE TABLE stats_per_backend_data(id int);
> + CREATE TABLE stats_per_backend_saved(wal_records bigint);
> +}
> +
> +teardown
> +{
> + DROP TABLE stats_per_backend_data;
> + DROP TABLE stats_per_backend_saved;
> +}
> +
> +session s1
> +setup { SET stats_fetch_consistency = 'none'; }
> +
> +step s1_fetch_consistency_cache { SET stats_fetch_consistency = 'cache'; }
> +step s1_fetch_consistency_snapshot { SET stats_fetch_consistency = 'snapshot'; }
> +step s1_begin { BEGIN; }
> +step s1_commit { COMMIT; }
> +step s1_build_snapshot {
> + SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal;
> +}
> +step s1_check_snapshot_backend {
> + SELECT wal_records = 0 AS snapshot_excludes_later_update
> + FROM pg_stat_get_backend_wal(
> + (SELECT pid FROM pg_stat_activity
> + WHERE application_name = 'isolation/stats-per-backend/s2'));
> +}
It's imo not particularly safe to assert 0, you can get on-access pruning or
hint-bit sets triggering WAL writes. I think you pretty much need to assert
that it's monotonically increasing, unfortunately.
> --- a/src/test/modules/test_misc/meson.build
> +++ b/src/test/modules/test_misc/meson.build
> @@ -24,6 +24,7 @@ tests += {
> 't/013_temp_obj_multisession.pl',
> 't/014_log_statement_max_length.pl',
> 't/015_temp_schema_exit_deferrable.pl',
> + 't/016_stats_snapshot.pl',
> ],
> # The injection points are cluster-wide, so disable installcheck
> 'runningcheck': false,
> diff --git a/src/test/modules/test_misc/t/016_stats_snapshot.pl b/src/test/modules/test_misc/t/016_stats_snapshot.pl
> new file mode 100644
> index 00000000000..43c9157bbb4
> --- /dev/null
> +++ b/src/test/modules/test_misc/t/016_stats_snapshot.pl
> @@ -0,0 +1,48 @@
> +# Copyright (c) 2026, PostgreSQL Global Development Group
> +
> +# Verify that a statistics snapshot excludes backends that started after the
> +# snapshot was created.
> +
> +use strict;
> +use warnings FATAL => 'all';
> +
> +use PostgreSQL::Test::Cluster;
> +use PostgreSQL::Test::Utils;
> +use Test::More;
> +
> +my $node = PostgreSQL::Test::Cluster->new('stats_snapshot');
> +$node->init;
> +$node->start;
Is this really worth creating another cluster for? That's not that cheap, and
our test codes really are exploding lately, largely due to creating lots of
tiny clusters that are just used for a tiny test.
> From 94c8483511c29e5837fbd299294ecb42d8cac265 Mon Sep 17 00:00:00 2001
> From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
> Date: Wed, 29 Jul 2026 14:11:27 +0000
> Subject: [PATCH v3 2/6] pgstat: add new infrastructure for per-backend
> statistics
>
> Add a new infrastructure for per-backend statistics that keep one shared entry
> per live backend while keeping their existing shared data as global stats for
> exited backends.
>
> Define a common dshash entry header containing the ProcNumber key, BackendType,
> and content LWLock. Extend PgStat_KindInfo with per-backend statistics related
> informations.
>
> At shared-memory initialization, create a dshash for every participating kind.
> +/* -------------------------------------------------------------------------
> + *
> + * pgstat_per_backend.c
> + * Generic infrastructure for per-backend statistics.
> + *
> + * This file manages the dedicated per-kind dshashes used for per-backend
> + * statistics, including entry creation, fetching, snapshots, transfer to
> + * global statistics, and removal.
It's not entirely obvious to me that a hash table is needed here. If I
understand correctly, this just going to be used for per-backend versions of
fixed stats. Which means we could just as well allocate all the memory in
statically allocated shared memory at server start and have the per-backend
stats be accessible by nothing more than an array access?
Given this is going to be used for stuff like WAL accesses etc, which just
about every backend will participate, I think the dshash overhead will just
cost (CPU, complexity, even memory), without gaining you meaningful memory
savings in realistic cases.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-10 14:45 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-13 15:41 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 07:15 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 09:17 ` Re: Redesign per-backend statistics Andres Freund <andres@anarazel.de>
@ 2026-09-22 00:16 ` Michael Paquier <michael@paquier.xyz>
2026-09-22 03:15 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 11+ messages in thread
From: Michael Paquier @ 2026-09-22 00:16 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; pgsql-hackers@lists.postgresql.org, Sami Imseih <samimseih@gmail.com>
On Mon, Sep 21, 2026 at 05:17:04AM -0400, Andres Freund wrote:
> On 2026-09-21 07:15:44 +0000, Bertrand Drouvot wrote:
>> +/* -------------------------------------------------------------------------
>> + *
>> + * pgstat_per_backend.c
>> + * Generic infrastructure for per-backend statistics.
>> + *
>> + * This file manages the dedicated per-kind dshashes used for per-backend
>> + * statistics, including entry creation, fetching, snapshots, transfer to
>> + * global statistics, and removal.
>
> It's not entirely obvious to me that a hash table is needed here. If I
> understand correctly, this just going to be used for per-backend versions of
> fixed stats. Which means we could just as well allocate all the memory in
> statically allocated shared memory at server start and have the per-backend
> stats be accessible by nothing more than an array access?
Yeah. Array access based on an procnum index is still feeling like
the natural thing to do here (I think I've mentioned that upthread?).
It's cheap to initialize, cheaper to access for reads and writes of
the data than one or two hash tables. For hot paths, that would
matter.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../arHI3tQDbpnd2El2@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-10 14:45 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-13 15:41 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 07:15 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 09:17 ` Re: Redesign per-backend statistics Andres Freund <andres@anarazel.de>
2026-09-22 00:16 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
@ 2026-09-22 03:15 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
0 siblings, 0 replies; 11+ messages in thread
From: Bertrand Drouvot @ 2026-09-22 03:15 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@lists.postgresql.org, Sami Imseih <samimseih@gmail.com>
Hi,
On Tue, Sep 22, 2026 at 09:16:30AM +0900, Michael Paquier wrote:
> On Mon, Sep 21, 2026 at 05:17:04AM -0400, Andres Freund wrote:
> > On 2026-09-21 07:15:44 +0000, Bertrand Drouvot wrote:
> >> +/* -------------------------------------------------------------------------
> >> + *
> >> + * pgstat_per_backend.c
> >> + * Generic infrastructure for per-backend statistics.
> >> + *
> >> + * This file manages the dedicated per-kind dshashes used for per-backend
> >> + * statistics, including entry creation, fetching, snapshots, transfer to
> >> + * global statistics, and removal.
> >
> > It's not entirely obvious to me that a hash table is needed here. If I
> > understand correctly, this just going to be used for per-backend versions of
> > fixed stats. Which means we could just as well allocate all the memory in
> > statically allocated shared memory at server start and have the per-backend
> > stats be accessible by nothing more than an array access?
>
> Yeah. Array access based on an procnum index is still feeling like
> the natural thing to do here (I think I've mentioned that upthread?).
Yeah you did mention it. I took your reply in [1] as agreeing with the concern
about reserving memory for mostly unused slots, though not necessarily with using
a dshash. Is your preference still for the array despite that memory cost?
[1]: https://postgr.es/m/anlVx_TD_Q8He5uf%40paquier.xyz
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Redesign per-backend statistics
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Re: Redesign per-backend statistics Michael Paquier <michael@paquier.xyz>
2026-08-10 14:45 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-13 15:41 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 07:15 ` Re: Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 09:17 ` Re: Redesign per-backend statistics Andres Freund <andres@anarazel.de>
@ 2026-09-22 03:07 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 0 replies; 11+ messages in thread
From: Bertrand Drouvot @ 2026-09-22 03:07 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers@lists.postgresql.org, Sami Imseih <samimseih@gmail.com>
Hi,
On Mon, Sep 21, 2026 at 05:17:04AM -0400, Andres Freund wrote:
> Hi,
>
> On 2026-09-21 07:15:44 +0000, Bertrand Drouvot wrote:
> > Subject: [PATCH v3 1/6] pgstat: add tests for per-backend statistics
>
> Good idea to start with that.
Thanks for looking at it!
> > Add an isolation test for per-backend WAL statistics. Verify that SNAPSHOT, CACHE
> > and NONE modes behave as expected.
>
> This isn't really related to this patch, but I do wonder if we need to make
> this somehow a bit more scalable for testing. If every stats thing has to test
> all of these "manually", we have a test-scalability issue...
Yeah, the idea is not to repeat these tests for each statistics kind. I had in
mind to use WAL as a representative kind to test the consistency behavior.
> > Extend test_shm_mq with a shared-memory-only worker that reports WAL usage
> > through a routine nonblocking flush and remains alive while the controlling
> > backend verifies that the global WAL counters include it.
>
> I'm not entirely sure what this really tests? Couldn't you just test this
> e.g. for checkpointer, in a quarter of the lines at most?
A background worker can access shared memory without establishing a database
connection, and therefore without calling InitPostgres(). The intent is to verify
that such a worker gets its per-backend entries during initialization and can
publish its statistics through the nonblocking path while remaining alive.
The nonblocking path is important because it cannot create a missing per-backend entry:
it succeeds only if pgstat_initialize() has already created and cached it. A forced
report could create the entry on demand and hide exactly an initialization bug
this test is intended to catch.
The checkpointer and worker_spi use forced reports, while logical replication workers
establish a database connection and can later force a report. Thus, none covers
this case.
Do you think this justifies the additional test code?
> It's imo not particularly safe to assert 0, you can get on-access pruning or
> hint-bit sets triggering WAL writes. I think you pretty much need to assert
> that it's monotonically increasing, unfortunately.
Yeah, will do in a next version, thanks.
> Is this really worth creating another cluster for? That's not that cheap, and
> our test codes really are exploding lately, largely due to creating lots of
> tiny clusters that are just used for a tiny test.
The test needs a process to start after the statistics snapshot has been built.
The isolation tester creates all its sessions before running any steps, so it
cannot cover this case.
I don't see an existing TAP test where this would fit naturally, and adding it to
an unrelated one only to reuse its node does not seem great either.
I think the coverage is useful because it verifies that a snapshot does not
fabricate entries for processes that did not exist when it was built.
Do you think this justifies the dedicated cluster, or would you prefer dropping
this test?
> It's not entirely obvious to me that a hash table is needed here. If I
> understand correctly, this just going to be used for per-backend versions of
> fixed stats. Which means we could just as well allocate all the memory in
> statically allocated shared memory at server start and have the per-backend
> stats be accessible by nothing more than an array access?
>
> Given this is going to be used for stuff like WAL accesses etc, which just
> about every backend will participate, I think the dshash overhead will just
> cost (CPU, complexity, even memory), without gaining you meaningful memory
> savings in realistic cases.
Yeah, each live process is expected to have an entry for these kinds.
The distinction though is between live processes and MaxBackends. A static array
has to be sized for every possible ProcNumber at startup, even if most slots are
never used.
If nearly all MaxBackends slots are live, then I agree that an array would likely
be preferable. The case I was trying to avoid is MaxBackends being substantially
higher than the usual number of live processes.
An earlier POC used such arrays. With max_connections=10000, it reserved about
33.5MB for the three kinds regardless of the number of live processes.
Global queries also had to scan unused slots or would require another structure
to track active ones.
The dshash allocates entries only for processes that actually exist. Also, each
process caches its entries during initialization, so the flush path does not
perform a hash lookup.
Does that explanation make sense, or do you still think a fixed array would be
preferable?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2026-09-22 03:15 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-08-03 14:20 Redesign per-backend statistics Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-07 01:39 ` Michael Paquier <michael@paquier.xyz>
2026-08-07 12:31 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-10 04:38 ` Michael Paquier <michael@paquier.xyz>
2026-08-10 14:45 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-08-13 15:41 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 07:15 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-21 09:17 ` Andres Freund <andres@anarazel.de>
2026-09-22 00:16 ` Michael Paquier <michael@paquier.xyz>
2026-09-22 03:15 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-09-22 03:07 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox