agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v10 5/7] Row pattern recognition patch (docs). 12+ messages / 2 participants [nested] [flat]
* [PATCH v10 5/7] Row pattern recognition patch (docs). @ 2023-10-22 02:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Tatsuo Ishii @ 2023-10-22 02:22 UTC (permalink / raw) --- doc/src/sgml/advanced.sgml | 80 ++++++++++++++++++++++++++++++++++++ doc/src/sgml/func.sgml | 54 ++++++++++++++++++++++++ doc/src/sgml/ref/select.sgml | 38 ++++++++++++++++- 3 files changed, 170 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml index 755c9f1485..cf18dd887e 100644 --- a/doc/src/sgml/advanced.sgml +++ b/doc/src/sgml/advanced.sgml @@ -537,6 +537,86 @@ WHERE pos < 3; <literal>rank</literal> less than 3. </para> + <para> + Row pattern common syntax can be used to perform row pattern recognition + in a query. Row pattern common syntax includes two sub + clauses: <literal>DEFINE</literal> + and <literal>PATTERN</literal>. <literal>DEFINE</literal> defines + definition variables along with an expression. The expression must be a + logical expression, which means it must + return <literal>TRUE</literal>, <literal>FALSE</literal> + or <literal>NULL</literal>. The expression may comprise column references + and functions. Window functions, aggregate functions and subqueries are + not allowed. An example of <literal>DEFINE</literal> is as follows. + +<programlisting> +DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +</programlisting> + + Note that <function>PREV</function> returns the price column in the + previous row if it's called in a context of row pattern recognition. So in + the second line the definition variable "UP" is <literal>TRUE</literal> + when the price column in the current row is greater than the price column + in the previous row. Likewise, "DOWN" is <literal>TRUE</literal> when when + the price column in the current row is lower than the price column in the + previous row. + </para> + <para> + Once <literal>DEFINE</literal> exists, <literal>PATTERN</literal> can be + used. <literal>PATTERN</literal> defines a sequence of rows that satisfies + certain conditions. For example following <literal>PATTERN</literal> + defines that a row starts with the condition "LOWPRICE", then one or more + rows satisfy "UP" and finally one or more rows satisfy "DOWN". Note that + "+" means one or more matches. Also you can use "*", which means zero or + more matches. If a sequence of rows which satisfies the PATTERN is found, + in the starting row of the sequence of rows all window functions and + aggregates are shown in the target list. Note that aggregations only look + into the matched rows, rather than whole frame. In the second or + subsequent rows all window functions and aggregates are NULL. For rows + that do not match the PATTERN, all window functions and aggregates are + shown AS NULL too, except count which shows 0. This is because the + unmatched rows are in an empty frame. Example of + a <literal>SELECT</literal> using the <literal>DEFINE</literal> + and <literal>PATTERN</literal> clause is as follows. + +<programlisting> +SELECT company, tdate, price, + first_value(price) OVER w, + max(price) OVER w, + count(price) OVER w +FROM stock, + WINDOW w AS ( + PARTITION BY company + ORDER BY tdate + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP PAST LAST ROW + INITIAL + PATTERN (LOWPRICE UP+ DOWN+) + DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +); +</programlisting> +<screen> + company | tdate | price | first_value | max | count +----------+------------+-------+-------------+-----+------- + company1 | 2023-07-01 | 100 | 100 | 200 | 4 + company1 | 2023-07-02 | 200 | | | + company1 | 2023-07-03 | 150 | | | + company1 | 2023-07-04 | 140 | | | + company1 | 2023-07-05 | 150 | | | 0 + company1 | 2023-07-06 | 90 | 90 | 130 | 4 + company1 | 2023-07-07 | 110 | | | + company1 | 2023-07-08 | 130 | | | + company1 | 2023-07-09 | 120 | | | + company1 | 2023-07-10 | 130 | | | 0 +</screen> + </para> + <para> When a query involves multiple window functions, it is possible to write out each one with a separate <literal>OVER</literal> clause, but this is diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 7c3e940afe..1d835af15a 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21878,6 +21878,7 @@ SELECT count(*) FROM sometable; returns <literal>NULL</literal> if there is no such row. </para></entry> </row> + </tbody> </tgroup> </table> @@ -21917,6 +21918,59 @@ SELECT count(*) FROM sometable; Other frame specifications can be used to obtain other effects. </para> + <para> + Row pattern recognition navigation functions are listed in + <xref linkend="functions-rpr-navigation-table"/>. These functions + can be used to describe DEFINE clause of Row pattern recognition. + </para> + + <table id="functions-rpr-navigation-table"> + <title>Row Pattern Navigation Functions</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="func_table_entry"><para role="func_signature"> + Function + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>prev</primary> + </indexterm> + <function>prev</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the previous row; + returns NULL if there is no previous row in the window frame. + </para></entry> + </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>next</primary> + </indexterm> + <function>next</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the next row; + returns NULL if there is no next row in the window frame. + </para></entry> + </row> + + </tbody> + </tgroup> + </table> + <note> <para> The SQL standard defines a <literal>RESPECT NULLS</literal> or diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml index 42d78913cf..522ad9dd70 100644 --- a/doc/src/sgml/ref/select.sgml +++ b/doc/src/sgml/ref/select.sgml @@ -969,8 +969,8 @@ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceabl The <replaceable class="parameter">frame_clause</replaceable> can be one of <synopsis> -{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] -{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] +{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] +{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] </synopsis> where <replaceable>frame_start</replaceable> @@ -1077,6 +1077,40 @@ EXCLUDE NO OTHERS a given peer group will be in the frame or excluded from it. </para> + <para> + The + optional <replaceable class="parameter">row_pattern_common_syntax</replaceable> + defines the <firstterm>row pattern recognition condition</firstterm> for + this + window. <replaceable class="parameter">row_pattern_common_syntax</replaceable> + includes following subclauses. <literal>AFTER MATCH SKIP PAST LAST + ROW</literal> or <literal>AFTER MATCH SKIP TO NEXT ROW</literal> controls + how to proceed to next row position after a match + found. With <literal>AFTER MATCH SKIP PAST LAST ROW</literal> (the + default) next row position is next to the last row of previous match. On + the other hand, with <literal>AFTER MATCH SKIP TO NEXT ROW</literal> next + row position is always next to the last row of previous + match. <literal>DEFINE</literal> defines definition variables along with a + boolean expression. <literal>PATTERN</literal> defines a sequence of rows + that satisfies certain conditions using variables defined + in <literal>DEFINE</literal> clause. If the variable is not defined in + the <literal>DEFINE</literal> clause, it is implicitly assumed + following is defined in the <literal>DEFINE</literal> clause. + +<synopsis> +<literal>variable_name</literal> AS TRUE +</synopsis> + + Note that the maximu number of variables defined + in <literal>DEFINE</literal> clause is 26. + +<synopsis> +[ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ] +PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...] +DEFINE <replaceable class="parameter">definition_varible_name</replaceable> AS <replaceable class="parameter">expression</replaceable> [, ...] +</synopsis> + </para> + <para> The purpose of a <literal>WINDOW</literal> clause is to specify the behavior of <firstterm>window functions</firstterm> appearing in the query's -- 2.25.1 ----Next_Part(Sun_Oct_22_11_39_20_2023_140)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v10-0006-Row-pattern-recognition-patch-tests.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v7 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 112 ++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 16 +++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 264 insertions(+), 66 deletions(-) 10.8% src/backend/postmaster/ 6.0% src/backend/replication/ 50.7% src/backend/utils/activity/ 6.0% src/backend/ 19.3% src/include/utils/ 5.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 2d8f57099fd..9f8ef8159d1 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -249,7 +251,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -271,6 +273,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 23e79a32345..ded0f250288 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -106,7 +108,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -116,6 +118,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 21de158adbb..2089de782d5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..a4ff64dc5ce 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +784,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..b340a680614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,21 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --C2xzVbxFmVrP7k6O Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v10 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 121 +++++++++++++++--- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 22 ++++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 279 insertions(+), 66 deletions(-) 10.5% src/backend/postmaster/ 5.8% src/backend/replication/ 51.0% src/backend/utils/activity/ 5.8% src/backend/ 18.7% src/include/utils/ 6.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13cce9b49f1..cf29fc91f70 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 7c1b8757d7d..aecc7a127e6 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index d01a09dd0c4..8c30efa2443 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ddd331e2c81 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -108,10 +108,12 @@ #include "pgstat.h" #include "storage/fd.h" #include "storage/ipc.h" +#include "storage/latch.h" #include "storage/lwlock.h" #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +124,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -218,6 +219,12 @@ PgStat_LocalState pgStatLocal; */ bool pgstat_report_fixed = false; +/* + * Track when there is pending anytime flush to avoid relying on + * get_timeout_active() in hot pathes. + */ +bool pgstat_pending_anytime = false; + /* ---------- * Local data * @@ -288,6 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +330,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +346,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +364,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +382,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +449,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +467,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +485,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +791,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1297,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1350,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1387,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1426,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); + + pgstat_pending_anytime = false; +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 3277cf88a4e..6b6eec7578d 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -117,11 +117,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 9bb777c3d5a..b011a315679 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -34,6 +34,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -532,8 +535,24 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !pgstat_pending_anytime) \ + { \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + pgstat_pending_anytime = true; \ + } \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); @@ -806,6 +825,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void); * Variables in pgstat.c */ +extern PGDLLIMPORT bool pgstat_pending_anytime; + /* GUC parameters */ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; @@ -849,4 +870,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime; /* updated by the traffic cop and in errfinish() */ extern PGDLLIMPORT SessionEndType pgStatSessionEndCause; + #endif /* PGSTAT_H */ diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index da28afbd929..4c207611236 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -84,7 +84,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -151,7 +151,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --NVvBxFuyV/R+1/8/ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v10-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v1 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushBehavior enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 18 +++++ src/backend/utils/activity/pgstat.c | 119 ++++++++++++++++++++++++---- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 11 +++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 154 insertions(+), 17 deletions(-) 9.7% src/backend/tcop/ 70.2% src/backend/utils/activity/ 9.3% src/backend/utils/init/ 6.0% src/include/utils/ 4.3% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..6a91543f80a 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,24 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + /* Skip if completely idle */ + if (!DoingCommandRead || IsTransactionOrTransactionBlock()) + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_ANYTIME_FLUSH_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..f7942e47475 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_behavior = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +395,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +412,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +446,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +464,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +482,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +788,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1346,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1383,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1397,11 +1417,42 @@ pgstat_flush_pending_entries(bool nowait) cur = next; } - Assert(dlist_is_empty(&pgStatPending) == !have_pending); + /* + * When in anytime_only mode, the list may not be empty because + * FLUSH_AT_TXN_BOUNDARY entries were skipped. + */ + Assert(!anytime_only || dlist_is_empty(&pgStatPending) == !have_pending); return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,37 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flush non-transactional stats + * + * This is safe to call even inside a transaction. It only flushes stats + * kinds marked as FLUSH_ANYTIME. + * + * This allows long running transactions to report activity without waiting + * for transaction to finish. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..cb0f6aecad1 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_ANYTIME_FLUSH_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..86e65397614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* When to call pgstat_report_anytime_stat() again */ +#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..02f4f13fc0f 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,14 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush behavior for statistics kinds. + */ +typedef enum PgStat_FlushBehavior +{ + FLUSH_ANYTIME, /* All fields can flush anytime */ + FLUSH_AT_TXN_BOUNDARY, /* All fields need transaction boundary */ +} PgStat_FlushBehavior; /* * Metadata for a specific kind of statistics. @@ -251,6 +259,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush behavior */ + PgStat_FlushBehavior flush_behavior; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 09e7f1d420e..9aabb325f16 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2261,6 +2261,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushBehavior PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --tjEWjIIwfNIHQLgt Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v1-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v3 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 111 +++++++++++++++++++++++----- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 16 ++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 148 insertions(+), 18 deletions(-) 8.1% src/backend/tcop/ 68.4% src/backend/utils/activity/ 9.6% src/backend/utils/init/ 9.2% src/include/utils/ 4.1% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..132fae61423 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_MIN_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ab4d9088a9a 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -122,8 +122,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +185,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +322,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +338,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +356,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +374,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +393,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +410,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +427,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +444,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +462,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +480,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +786,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1344,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1381,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1402,6 +1420,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2164,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..6076f531c4a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..1651f16f966 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..46ce90c9624 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush mode */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3f3a888fd0e..d3912b43fdc 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --lCc1MWfC7i2uqs8g Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v5 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 9 ++ src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 118 ++++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 9 ++ src/backend/utils/activity/pgstat_io.c | 5 + src/backend/utils/activity/pgstat_slru.c | 5 + src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 21 ++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 19 files changed, 213 insertions(+), 25 deletions(-) 5.5% src/backend/access/transam/ 15.3% src/backend/postmaster/ 3.4% src/backend/replication/ 3.8% src/backend/tcop/ 57.0% src/backend/utils/activity/ 9.2% src/include/utils/ 5.3% src/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..9503aea5b4d 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,10 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2070,11 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_MIN_INTERVAL); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 80e3088fc7e..ab5d0645026 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -104,7 +106,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -114,6 +116,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index 6482c21b8f9..6e187315613 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -216,7 +217,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -226,6 +227,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index a1a4f65f9a9..498d147f0da 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -246,6 +247,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index c3d56c866d3..cec5dfdb430 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -250,7 +252,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -272,6 +274,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 38ec8a4c8c7..7416ca703c9 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -107,7 +109,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -117,6 +119,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 8b99160ed0e..24d7ef795cb 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -253,7 +255,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -261,6 +263,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index b4a8d2f3a1c..d19aa45400d 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..2c9454677e9 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +394,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +411,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +428,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +445,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +463,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +481,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +787,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,12 +1293,18 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); entry_ref->pending = MemoryContextAllocZero(pgStatPendingContext, entrysize); dlist_push_tail(&pgStatPending, &entry_ref->pending_node); + + /* Schedule next anytime stats update timeout */ + if (kind_info->flush_mode == FLUSH_ANYTIME && IsUnderPostmaster && + !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); } return entry_ref; @@ -1345,9 +1351,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1388,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1402,6 +1427,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2171,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing non-transactional stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index 1350f5f62f1..9dcb24db975 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,10 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +87,10 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + backend_has_iostats = true; pgstat_report_fixed = true; } diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..53dbf2a514b 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,10 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + have_iostats = true; pgstat_report_fixed = true; } diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..1d16cde1889 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -223,6 +224,10 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..f45365f47f7 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -765,6 +766,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..1651f16f966 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..a9190078d0e 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,13 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * Some stats have to be updated only at transaction boundaries (such as + * tuples_inserted updated, deleted), so it's very important to set the + * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default). + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -677,6 +697,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 9f5ee8fd482..860f835c088 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --WLtTrIUEfQDmERxy Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v6 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 112 ++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 16 +++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 264 insertions(+), 66 deletions(-) 10.8% src/backend/postmaster/ 6.0% src/backend/replication/ 50.7% src/backend/utils/activity/ 6.0% src/backend/ 19.3% src/include/utils/ 5.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 2d8f57099fd..9f8ef8159d1 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -249,7 +251,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -271,6 +273,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 23e79a32345..ded0f250288 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -106,7 +108,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -116,6 +118,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 02e9aaa6bca..c7bc409b06f 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3509,6 +3509,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..411b65aae3e 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +784,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing non-transactional stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index 1350f5f62f1..b0bd220882d 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index d7f6d4c5ee6..61094f96a6c 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -425,7 +425,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..eb5f8f46925 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..f45365f47f7 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -765,6 +766,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..b340a680614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,21 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 9f5ee8fd482..860f835c088 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --DxqFthphbM+ha9Dy Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v8 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 112 ++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 16 +++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 264 insertions(+), 66 deletions(-) 10.8% src/backend/postmaster/ 6.0% src/backend/replication/ 50.7% src/backend/utils/activity/ 6.0% src/backend/ 19.3% src/include/utils/ 5.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 21de158adbb..2089de782d5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..a4ff64dc5ce 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +784,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..b340a680614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,21 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --OI5irqZsxWxBW9nT Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v8-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v9 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 120 +++++++++++++++--- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 22 ++++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 278 insertions(+), 66 deletions(-) 10.5% src/backend/postmaster/ 5.8% src/backend/replication/ 50.9% src/backend/utils/activity/ 5.9% src/backend/ 18.8% src/include/utils/ 6.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 21de158adbb..2089de782d5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..419dc512d9b 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -218,6 +218,12 @@ PgStat_LocalState pgStatLocal; */ bool pgstat_report_fixed = false; +/* + * Track when there is pending anytime flush to avoid relying on + * get_timeout_active() in hot pathes. + */ +bool pgstat_pending_anytime = false; + /* ---------- * Local data * @@ -288,6 +294,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +312,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +329,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +345,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +363,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +381,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +448,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +466,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +484,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +790,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1296,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1349,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1386,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1425,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2169,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); + + pgstat_pending_anytime = false; +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..f0f546d419a 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,24 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !pgstat_pending_anytime) \ + { \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + pgstat_pending_anytime = true; \ + } \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); @@ -808,6 +827,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void); * Variables in pgstat.c */ +extern PGDLLIMPORT bool pgstat_pending_anytime; + /* GUC parameters */ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; @@ -851,4 +872,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime; /* updated by the traffic cop and in errfinish() */ extern PGDLLIMPORT SessionEndType pgStatSessionEndCause; + #endif /* PGSTAT_H */ diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --aq/6bi8L6WORnI+9 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v11 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 121 +++++++++++++++--- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 22 ++++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 279 insertions(+), 66 deletions(-) 10.5% src/backend/postmaster/ 5.8% src/backend/replication/ 51.0% src/backend/utils/activity/ 5.8% src/backend/ 18.7% src/include/utils/ 6.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13cce9b49f1..cf29fc91f70 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 7c1b8757d7d..aecc7a127e6 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index d01a09dd0c4..8c30efa2443 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ddd331e2c81 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -108,10 +108,12 @@ #include "pgstat.h" #include "storage/fd.h" #include "storage/ipc.h" +#include "storage/latch.h" #include "storage/lwlock.h" #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +124,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -218,6 +219,12 @@ PgStat_LocalState pgStatLocal; */ bool pgstat_report_fixed = false; +/* + * Track when there is pending anytime flush to avoid relying on + * get_timeout_active() in hot pathes. + */ +bool pgstat_pending_anytime = false; + /* ---------- * Local data * @@ -288,6 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +330,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +346,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +364,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +382,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +449,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +467,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +485,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +791,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1297,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1350,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1387,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1426,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); + + pgstat_pending_anytime = false; +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 3277cf88a4e..6b6eec7578d 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -117,11 +117,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 9bb777c3d5a..b011a315679 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -34,6 +34,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -532,8 +535,24 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !pgstat_pending_anytime) \ + { \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + pgstat_pending_anytime = true; \ + } \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); @@ -806,6 +825,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void); * Variables in pgstat.c */ +extern PGDLLIMPORT bool pgstat_pending_anytime; + /* GUC parameters */ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; @@ -849,4 +870,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime; /* updated by the traffic cop and in errfinish() */ extern PGDLLIMPORT SessionEndType pgStatSessionEndCause; + #endif /* PGSTAT_H */ diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index da28afbd929..4c207611236 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -84,7 +84,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -151,7 +151,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --2MEBAGW8+kohXisi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v11-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v2 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushBehavior enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 113 ++++++++++++++++++++++++---- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 13 ++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 149 insertions(+), 16 deletions(-) 8.2% src/backend/tcop/ 69.3% src/backend/utils/activity/ 9.7% src/backend/utils/init/ 7.7% src/include/utils/ 4.5% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..9c4a9078ee0 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_ANYTIME_FLUSH_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..0f45a7d165e 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_behavior = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +395,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +412,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +446,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +464,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +482,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +788,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1346,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1383,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1397,11 +1417,42 @@ pgstat_flush_pending_entries(bool nowait) cur = next; } + /* + * When in anytime_only mode, the list may not be empty because + * FLUSH_AT_TXN_BOUNDARY entries were skipped. + */ Assert(dlist_is_empty(&pgStatPending) == !have_pending); return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..cb0f6aecad1 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_ANYTIME_FLUSH_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..86e65397614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* When to call pgstat_report_anytime_stat() again */ +#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..63feae640d1 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,16 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush behavior for statistics kinds. + */ +typedef enum PgStat_FlushBehavior +{ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ +} PgStat_FlushBehavior; /* * Metadata for a specific kind of statistics. @@ -251,6 +261,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush behavior */ + PgStat_FlushBehavior flush_behavior; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3f3a888fd0e..610b35a9b31 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushBehavior PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --OFsrIl+bjhifp5hk Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v4 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/storage/lmgr/proc.c | 10 +++ src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 111 +++++++++++++++++++++++----- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 20 +++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 10 files changed, 162 insertions(+), 18 deletions(-) 7.0% src/backend/storage/lmgr/ 7.3% src/backend/tcop/ 61.1% src/backend/utils/activity/ 8.6% src/backend/utils/init/ 11.8% src/include/utils/ 3.6% src/include/ diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c index 063826ae576..012705a2ee6 100644 --- a/src/backend/storage/lmgr/proc.c +++ b/src/backend/storage/lmgr/proc.c @@ -1322,6 +1322,7 @@ ProcSleep(LOCALLOCK *locallock) bool allow_autovacuum_cancel = true; bool logged_recovery_conflict = false; ProcWaitStatus myWaitStatus; + bool anytime_timeout_was_active = false; /* The caller must've armed the on-error cleanup mechanism */ Assert(GetAwaitedLock() == locallock); @@ -1398,6 +1399,12 @@ ProcSleep(LOCALLOCK *locallock) standbyWaitStart = GetCurrentTimestamp(); } + anytime_timeout_was_active = get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT); + + /* No need to try to flush the statistics while the process is sleeping */ + if (anytime_timeout_was_active) + disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false); + /* * If somebody wakes us between LWLockRelease and WaitLatch, the latch * will not wait. But a set latch does not necessarily mean that the lock @@ -1661,6 +1668,9 @@ ProcSleep(LOCALLOCK *locallock) } } while (myWaitStatus == PROC_WAIT_STATUS_WAITING); + if (anytime_timeout_was_active) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + /* * Disable the timers, if they are still running. As in LockErrorCleanup, * we must preserve the LOCK_TIMEOUT indicator flag: if a lock timeout has diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..132fae61423 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_MIN_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ab4d9088a9a 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -122,8 +122,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +185,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +322,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +338,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +356,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +374,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +393,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +410,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +427,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +444,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +462,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +480,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +786,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1344,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1381,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1402,6 +1420,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2164,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..6076f531c4a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..1651f16f966 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..9ca39ea9a9a 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,13 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * Some stats have to be updated only at transaction boundaries (such as + * tuples_inserted updated, deleted), so it's very important to set the + * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default). + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index ddbe4c64971..af21c87234a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --wR/mWGXukst4NraF Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2026-01-05 09:41 UTC | newest] Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-10-22 02:22 [PATCH v10 5/7] Row pattern recognition patch (docs). Tatsuo Ishii <[email protected]> 2026-01-05 09:41 [PATCH v7 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v10 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v1 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v3 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v5 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v6 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v8 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v9 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v11 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v2 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v4 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox