public inbox for [email protected]
help / color / mirror / Atom feedRe: Flush some statistics within running transactions
27+ messages / 6 participants
[nested] [flat]
* Re: Flush some statistics within running transactions
@ 2026-01-19 12:17 Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-20 23:18 ` Re: Flush some statistics within running transactions Zsolt Parragi <[email protected]>
0 siblings, 2 replies; 27+ messages in thread
From: Bertrand Drouvot @ 2026-01-19 12:17 UTC (permalink / raw)
To: Sami Imseih <[email protected]>; +Cc: [email protected]
Hi,
On Fri, Jan 16, 2026 at 10:44:48AM -0600, Sami Imseih wrote:
> I took a look at 0001 in depth.
Thanks!
> > I don't think this feature could add a noticeable performance impact, so the tests
> > have been that simple. Do you think we should worry more?
>
> One observation is there's no coordination between ANYTIME and
> TXN_BOUNDARY flushes. While PGSTAT_MIN_INTERVAL
> prevents a backend from flushing more than once per second, a backend can
> still perform both an ANYTIME flush and a TXN_BOUNDARY flush within
> the same 1-second window. Not saying this will be a real problem in
> the real-world,
> but we definitely took measures in the current implementation to avoid
> this scenario.
Right. I think that the PGSTAT_MIN_INTERVAL throttling was put in place to prevent
flushing too frequently when the backend has a high commit rate. But here, while
it's true that we don't follow that rule (means a backend could flush more than one
time per second), that would be a maximum of 2 times (given that ANYTIME is
flushing every second). So, I'm not sure that this single extra flush is worth
worrying about. Plus we'd certainly need an extra GetCurrentTimestamp() call, so
I'm not sure it's worth it.
> A few other comments on 0001
>
> + /* Skip if completely idle */
> + if (!DoingCommandRead || IsTransactionOrTransactionBlock())
> + pgstat_report_anytime_stat(false);
>
> Does this need to be conditional? worst case, we return right away with an empty
> list. Best case, is we are consistently flushing.
Yeah, I think we could remove this check and just rely on the ones in
pgstat_report_anytime_stat(). Done in the attached.
> + Assert(!anytime_only || dlist_is_empty(&pgStatPending) ==
> !have_pending);
>
> Checking for !anytime_only is unnecessary here.
> "list_is_empty(&pgStatPending) == !have_pending"
> should be true regardless of ANYTIME or TXN_BOUNDARY, right?
Right, thanks for catching it, it was remaining garbage from my dev iterations.
> Below are a couple of edits for comments I felt would improve
> readability of the code.
Done as suggested.
> I will start looking at the remaining patches next.
Thanks!
Note that I also updated the doc in 0003 for the stats that have mixed fields.
BTW, I think that we could also make the Function stat kind as flush any time,
thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-01-20 19:27 ` Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
1 sibling, 1 reply; 27+ messages in thread
From: Sami Imseih @ 2026-01-20 19:27 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: [email protected]
Thanks for the updates!
> > > I don't think this feature could add a noticeable performance impact, so the tests
> > > have been that simple. Do you think we should worry more?
> >
> > One observation is there's no coordination between ANYTIME and
> > TXN_BOUNDARY flushes. While PGSTAT_MIN_INTERVAL
> > prevents a backend from flushing more than once per second, a backend can
> > still perform both an ANYTIME flush and a TXN_BOUNDARY flush within
> > the same 1-second window. Not saying this will be a real problem in
> > the real-world,
> > but we definitely took measures in the current implementation to avoid
> > this scenario.
>
> Right. I think that the PGSTAT_MIN_INTERVAL throttling was put in place to prevent
> flushing too frequently when the backend has a high commit rate. But here, while
> it's true that we don't follow that rule (means a backend could flush more than one
> time per second), that would be a maximum of 2 times (given that ANYTIME is
> flushing every second). So, I'm not sure that this single extra flush is worth
> worrying about. Plus we'd certainly need an extra GetCurrentTimestamp() call, so
> I'm not sure it's worth it.
Yeah, all PGSTAT_MIN_INTERVAL does is throttle pgstat_flush_pending_entries.
Even in the current state, it does not limit how many kinds are flushed, etc.
I consider the ANYTIME flushes the same as just adding another stats kind.
So, I am not really worried about either.
I have some more comments:
-- v2-0001
#1.
+/* When to call pgstat_report_anytime_stat() again */
+#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000
+
We should just use PGSTAT_MIN_INTERVAL.
#2.
instead of ".flush_behavior", maybe ".flush_mode"? "mode" in the name is better
for configuration fields.
#3.
+/*
+ * Flush behavior for statistics kinds.
+ */
+typedef enum PgStat_FlushBehavior
+{
+ FLUSH_ANYTIME, /* All fields can be
flushed anytime,
+ *
including within transactions */
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ *
transaction boundary */
+} PgStat_FlushBehavior;
FLUSH_AT_TXN_BOUNDARY should be the first value in PgStat_FlushBehavior.
Otherwise kinds ( built-in or custom ) that do not specify a flush_behavior
will default to FLUSH_ANYTIME. I don't think this is what we want.
FLUSH_AT_TXN_BOUNDARY should be the default.
#4. Can we add a test here? Maybe generate some wal inside a long
running transaction and
make sure the stats are updated after > 1 second
-- v2-0002
No comments for this one. With ANYTIME, indeed those flushes are not needed.
-- v2-0003
#1. Should we maybe make this a bit longer? maybe 2 or 3 seconds?
May make the tests slightly longer, but maybe better for test stability.
```
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
```
#2.
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0)
+ has_nontxn_stats = true;
+
+ if (!has_nontxn_stats)
+ return true;
Can we just do this without a has_nontxn_stats?
This is also the same patter as a regular flush, although
in the case `pg_memory_is_all_zeros` is used.
```
if (lstats->counts.numscans == 0 &&
lstats->counts.tuples_returned == 0 &&
lstats->counts.tuples_fetched == 0 &&
lstats->counts.blocks_fetched == 0 &&
lstats->counts.blocks_hit == 0)
return true;
```
#3.
+ are updated while the transactions are in progress. This means
that we can see
+ those statistics being updated without having to wait until the transaction
+ finishes.
+ </para>
The "This means ...... " line used several times does not add value, IMO.
"are updated while the transactions are in progress." is sufficient.
#4.
+ <note>
+ <para>
+ All the statistics are updated while the transactions are in
progress, except
+ for <structfield>xact_commit</structfield>,
<structfield>xact_rollback</structfield>,
+ <structfield>tup_inserted</structfield>,
<structfield>tup_updated</structfield> and
+ <structfield>tup_deleted</structfield> that are updated only when
the transactions
+ finish.
+ </para>
+ </note>
Only these 5 fields from pgstat_relation_flush_anytime_cb, so only the below are
"All the statistics are updated while the transactions are in progress", right?
numscans
tuples_returned
tuples_fetched
blocks_fetched
blocks_hit
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
@ 2026-01-21 10:34 ` Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Bertrand Drouvot @ 2026-01-21 10:34 UTC (permalink / raw)
To: Sami Imseih <[email protected]>; +Cc: [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Tue, Jan 20, 2026 at 01:27:55PM -0600, Sami Imseih wrote:
> I have some more comments:
Thanks!
> -- v2-0001
>
> #1.
>
> +/* When to call pgstat_report_anytime_stat() again */
> +#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000
> +
>
> We should just use PGSTAT_MIN_INTERVAL.
Okay, done. We can still switch to a dedicated one if we feel the need later on.
> #2.
>
> instead of ".flush_behavior", maybe ".flush_mode"? "mode" in the name is better
> for configuration fields.
Sounds good.
> #3.
>
> FLUSH_AT_TXN_BOUNDARY should be the first value in PgStat_FlushBehavior.
> Otherwise kinds ( built-in or custom ) that do not specify a flush_behavior
> will default to FLUSH_ANYTIME. I don't think this is what we want.
> FLUSH_AT_TXN_BOUNDARY should be the default.
Good point, agreed and done.
> #4. Can we add a test here? Maybe generate some wal inside a long
> running transaction and
> make sure the stats are updated after > 1 second
I'm not sure, that's also somehow the purpose of 0002 (with 039549d70f6 being
reverted).
0001 and 0002 could be merged and pushed as one commit. That said I'm not opposed
if you feel strongly about it.
> -- v2-0003
>
> #1. Should we maybe make this a bit longer? maybe 2 or 3 seconds?
> May make the tests slightly longer, but maybe better for test stability.
>
> ```
> +step s1_sleep: SELECT pg_sleep(1.5);
> +pg_sleep
> +--------
> ```
Not sure, we could increase if we see the test failing.
> #2.
> + /*
> + * Check if there are any non-transactional stats to flush. Avoid
> + * unnecessarily locking the entry if nothing accumulated.
> + */
> + if (lstats->counts.numscans > 0 ||
> + lstats->counts.tuples_returned > 0 ||
> + lstats->counts.tuples_fetched > 0 ||
> + lstats->counts.blocks_fetched > 0 ||
> + lstats->counts.blocks_hit > 0)
> + has_nontxn_stats = true;
> +
> + if (!has_nontxn_stats)
> + return true;
> Can we just do this without a has_nontxn_stats?
Yeah.
> #3.
> + are updated while the transactions are in progress. This means
> that we can see
> + those statistics being updated without having to wait until the transaction
> + finishes.
> + </para>
>
> The "This means ...... " line used several times does not add value, IMO.
> "are updated while the transactions are in progress." is sufficient.
Removed.
>
> #4.
> + <note>
> + <para>
> + All the statistics are updated while the transactions are in
> progress, except
> + for <structfield>xact_commit</structfield>,
> <structfield>xact_rollback</structfield>,
> + <structfield>tup_inserted</structfield>,
> <structfield>tup_updated</structfield> and
> + <structfield>tup_deleted</structfield> that are updated only when
> the transactions
> + finish.
> + </para>
> + </note>
>
> Only these 5 fields from pgstat_relation_flush_anytime_cb, so only the below are
> "All the statistics are updated while the transactions are in progress", right?
>
> numscans
> tuples_returned
> tuples_fetched
> blocks_fetched
> blocks_hit
No, 0003 also changes the flush mode for the database KIND. All the fields that
I mentioned are inherited from relations stats and are flushed only at transaction
boundaries (so they don't appear in pg_stat_database until the transaction
finishes). Does that make sense? (if the database kind is not switched to
flush any time then none would appear while the transaction is in progress, even
the ones inherited from relations stats).
PFA v3, also taking care of Zsolt's comment (thanks!) done up-thread.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-01-22 00:02 ` Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Michael Paquier @ 2026-01-22 00:02 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Sami Imseih <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
On Wed, Jan 21, 2026 at 10:34:09AM +0000, Bertrand Drouvot wrote:
> No, 0003 also changes the flush mode for the database KIND. All the fields that
> I mentioned are inherited from relations stats and are flushed only at transaction
> boundaries (so they don't appear in pg_stat_database until the transaction
> finishes). Does that make sense? (if the database kind is not switched to
> flush any time then none would appear while the transaction is in progress, even
> the ones inherited from relations stats).
>
> PFA v3, also taking care of Zsolt's comment (thanks!) done up-thread.
While reading through 0001, I got to question on which properties
and/or assumptions of a stats kind one has to rely on to decide to
what flush_mode should be set. To put is simpler, why don't we just
do a periodic pgstat_report_stat(false) call that would flush all the
stats for all stats kinds based on the new timeout registered,
expanding a bit the flush we currently do when idle in
ProcessInterrupts()? It seems that one point of contention should be
that we should be careful with entries in the shmem hash table that
have been created in a transactional way, but we may already flush
them while we are in a transaction state, no? Are there any fields in
a stats kind that we do may not want to flush? If yes, it sounds to
me that it would be better to document these in the structures to
explain the reason why a flush mode is chosen over the other.
I am also not convinced that we have to be that aggressive with these
extra flushes. The target is long-running analytical queries, that
could take minutes or even hours. Using the same value as
PGSTAT_IDLE_INTERVAL (10s), perhaps renaming the value while on it,
would be a more natural fit. A 1s vs 10s report interval does not
really matter for long analytical queries, where I'd imagine data
being picked up on at least a 30s interval, at the shortest. Of
course, one may want to get a more "live" representation of the data
with more aggressive flushes, but is that really helpful for
long-running queries to have more granularity, stressing more the
shmem state?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, 2-signature.asc)
download
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
@ 2026-01-22 01:41 ` Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 02:28 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
0 siblings, 2 replies; 27+ messages in thread
From: Sami Imseih @ 2026-01-22 01:41 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
> > No, 0003 also changes the flush mode for the database KIND. All the fields that
> > I mentioned are inherited from relations stats and are flushed only at transaction
> > boundaries (so they don't appear in pg_stat_database until the transaction
> > finishes). Does that make sense? (if the database kind is not switched to
> > flush any time then none would appear while the transaction is in progress, even
> > the ones inherited from relations stats).
> >
> > PFA v3, also taking care of Zsolt's comment (thanks!) done up-thread.
>
> While reading through 0001, I got to question on which properties
> and/or assumptions of a stats kind one has to rely on to decide to
> what flush_mode should be set. To put is simpler, why don't we just
> do a periodic pgstat_report_stat(false) call that would flush all the
> stats for all stats kinds based on the new timeout registered,
> expanding a bit the flush we currently do when idle in
> ProcessInterrupts()?
There are some important cases in which we would want to
distinguish between a "transaction boundary" flush vs an
"anytime" flush.
For example, xact_commit/rollback. I would want those
fields to be in sync with tuples_inserted/updated/deleted
to allow for accurate calculations like number of inserts
per commit, etc.
Another one would be n_mod_since_analyze, That should
only be updated after commit (or not after rollback). Otherwise,
it may throw autovanalyze threshold calculations way off. Same
for n_dead_tup and autovacuum.
> I am also not convinced that we have to be that aggressive with these
> extra flushes. The target is long-running analytical queries, that
> could take minutes or even hours. Using the same value as
> PGSTAT_IDLE_INTERVAL (10s),
PGSTAT_IDLE_INTERVAL is flushing an idle backend every 10 seconds
IIUC. So this value only applies when outside of a transaction.
> A 1s vs 10s report interval does not really matter for long analytical queries.
Sure, Bertrand mentioned early in the thread that the anytime flushes
could be made configurable. Perhaps that is a good idea where we can
default with something large like 10s intervals for anytime flushes, but allow
the user to configure a more frequent flushes ( although I would think
that 1 sec is the minimum we should allow ).
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
@ 2026-01-22 01:56 ` Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
1 sibling, 1 reply; 27+ messages in thread
From: Fujii Masao @ 2026-01-22 01:56 UTC (permalink / raw)
To: Sami Imseih <[email protected]>; +Cc: Michael Paquier <[email protected]>; Bertrand Drouvot <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
On Thu, Jan 22, 2026 at 10:41 AM Sami Imseih <[email protected]> wrote:
>
> > > No, 0003 also changes the flush mode for the database KIND. All the fields that
> > > I mentioned are inherited from relations stats and are flushed only at transaction
> > > boundaries (so they don't appear in pg_stat_database until the transaction
> > > finishes). Does that make sense? (if the database kind is not switched to
> > > flush any time then none would appear while the transaction is in progress, even
> > > the ones inherited from relations stats).
> > >
> > > PFA v3, also taking care of Zsolt's comment (thanks!) done up-thread.
> >
> > While reading through 0001, I got to question on which properties
> > and/or assumptions of a stats kind one has to rely on to decide to
> > what flush_mode should be set. To put is simpler, why don't we just
> > do a periodic pgstat_report_stat(false) call that would flush all the
> > stats for all stats kinds based on the new timeout registered,
> > expanding a bit the flush we currently do when idle in
> > ProcessInterrupts()?
>
> There are some important cases in which we would want to
> distinguish between a "transaction boundary" flush vs an
> "anytime" flush.
>
> For example, xact_commit/rollback. I would want those
> fields to be in sync with tuples_inserted/updated/deleted
> to allow for accurate calculations like number of inserts
> per commit, etc.
>
> Another one would be n_mod_since_analyze, That should
> only be updated after commit (or not after rollback). Otherwise,
> it may throw autovanalyze threshold calculations way off. Same
> for n_dead_tup and autovacuum.
>
> > I am also not convinced that we have to be that aggressive with these
> > extra flushes. The target is long-running analytical queries, that
> > could take minutes or even hours. Using the same value as
> > PGSTAT_IDLE_INTERVAL (10s),
>
> PGSTAT_IDLE_INTERVAL is flushing an idle backend every 10 seconds
> IIUC. So this value only applies when outside of a transaction.
>
> > A 1s vs 10s report interval does not really matter for long analytical queries.
>
> Sure, Bertrand mentioned early in the thread that the anytime flushes
> could be made configurable. Perhaps that is a good idea where we can
> default with something large like 10s intervals for anytime flushes, but allow
> the user to configure a more frequent flushes ( although I would think
> that 1 sec is the minimum we should allow ).
+1 on adding an option to control the interval. With a fixed interval
(for example, 1s), log_lock_waits messages could be emitted that frequently,
which may be annoying for some users.
Of course, it would be even better if these periodic wakeups did not trigger
log_lock_waits messages at all, though.
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
@ 2026-01-22 07:43 ` Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Bertrand Drouvot @ 2026-01-22 07:43 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Sami Imseih <[email protected]>; Michael Paquier <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Thu, Jan 22, 2026 at 10:56:48AM +0900, Fujii Masao wrote:
> On Thu, Jan 22, 2026 at 10:41 AM Sami Imseih <[email protected]> wrote:
> >
> > Sure, Bertrand mentioned early in the thread that the anytime flushes
> > could be made configurable. Perhaps that is a good idea where we can
> > default with something large like 10s intervals for anytime flushes, but allow
> > the user to configure a more frequent flushes ( although I would think
> > that 1 sec is the minimum we should allow ).
>
> +1 on adding an option to control the interval. With a fixed interval
> (for example, 1s), log_lock_waits messages could be emitted that frequently,
> which may be annoying for some users.
>
> Of course, it would be even better if these periodic wakeups did not trigger
> log_lock_waits messages at all, though.
pgstat_report_anytime_stat() is called with the force parameter set to false,
means that the flushes are done with nowait = true means that LWLockConditionalAcquire()
is used. In that case, do you still see cases where log_lock_waits messages could
be triggered due to the new flush?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-01-22 12:12 ` Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-30 09:46 ` Re: Flush some statistics within running transactions Álvaro Herrera <[email protected]>
0 siblings, 2 replies; 27+ messages in thread
From: Fujii Masao @ 2026-01-22 12:12 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Sami Imseih <[email protected]>; Michael Paquier <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
On Thu, Jan 22, 2026 at 4:43 PM Bertrand Drouvot
<[email protected]> wrote:
>
> Hi,
>
> On Thu, Jan 22, 2026 at 10:56:48AM +0900, Fujii Masao wrote:
> > On Thu, Jan 22, 2026 at 10:41 AM Sami Imseih <[email protected]> wrote:
> > >
> > > Sure, Bertrand mentioned early in the thread that the anytime flushes
> > > could be made configurable. Perhaps that is a good idea where we can
> > > default with something large like 10s intervals for anytime flushes, but allow
> > > the user to configure a more frequent flushes ( although I would think
> > > that 1 sec is the minimum we should allow ).
> >
> > +1 on adding an option to control the interval. With a fixed interval
> > (for example, 1s), log_lock_waits messages could be emitted that frequently,
> > which may be annoying for some users.
> >
> > Of course, it would be even better if these periodic wakeups did not trigger
> > log_lock_waits messages at all, though.
>
> pgstat_report_anytime_stat() is called with the force parameter set to false,
> means that the flushes are done with nowait = true means that LWLockConditionalAcquire()
> is used. In that case, do you still see cases where log_lock_waits messages could
> be triggered due to the new flush?
I haven't read the patch in detail yet, but after applying patch 0001 and
causing a lock wait (for example, using the steps below), I observed that
log_lock_waits messages are emitted every second.
[session 1]
create table tbl as select id from generate_series(1, 10) id;
begin;
select * from tbl where id = 1 for update;
[session 2]
begin;
select * from tbl where id = 1 for update;
With this setup, the following messages were logged once per second:
LOG: process 72199 still waiting for ShareLock on transaction 771
after 63034.119 ms
DETAIL: Process holding the lock: 72190. Wait queue: 72199.
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
@ 2026-01-22 16:45 ` Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
1 sibling, 1 reply; 27+ messages in thread
From: Bertrand Drouvot @ 2026-01-22 16:45 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Sami Imseih <[email protected]>; Michael Paquier <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Thu, Jan 22, 2026 at 09:12:18PM +0900, Fujii Masao wrote:
> On Thu, Jan 22, 2026 at 4:43 PM Bertrand Drouvot
> <[email protected]> wrote:
> >
> > pgstat_report_anytime_stat() is called with the force parameter set to false,
> > means that the flushes are done with nowait = true means that LWLockConditionalAcquire()
> > is used. In that case, do you still see cases where log_lock_waits messages could
> > be triggered due to the new flush?
>
> I haven't read the patch in detail yet, but after applying patch 0001 and
> causing a lock wait (for example, using the steps below), I observed that
> log_lock_waits messages are emitted every second.
>
> [session 1]
> create table tbl as select id from generate_series(1, 10) id;
> begin;
> select * from tbl where id = 1 for update;
>
> [session 2]
> begin;
> select * from tbl where id = 1 for update;
>
> With this setup, the following messages were logged once per second:
>
> LOG: process 72199 still waiting for ShareLock on transaction 771
> after 63034.119 ms
> DETAIL: Process holding the lock: 72190. Wait queue: 72199.
>
Thanks!
I see, the WaitLatch() in ProcSleep() is "woken up" every 1s due to the
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,...) being set unconditionally
in ProcessInterrupts(). We need to be more restrictive as to when to enable the
timeout, I'll fix in the next version.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-01-26 06:59 ` Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Bertrand Drouvot @ 2026-01-26 06:59 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Sami Imseih <[email protected]>; Michael Paquier <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Thu, Jan 22, 2026 at 04:45:31PM +0000, Bertrand Drouvot wrote:
> On Thu, Jan 22, 2026 at 09:12:18PM +0900, Fujii Masao wrote:
> >
> > With this setup, the following messages were logged once per second:
> >
> > LOG: process 72199 still waiting for ShareLock on transaction 771
> > after 63034.119 ms
> > DETAIL: Process holding the lock: 72190. Wait queue: 72199.
> >
>
> Thanks!
>
> I see, the WaitLatch() in ProcSleep() is "woken up" every 1s due to the
> enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,...) being set unconditionally
> in ProcessInterrupts(). We need to be more restrictive as to when to enable the
> timeout, I'll fix in the next version.
The attached, to apply on top of 0001, fix the issue. However it handles only the
WaitLatch in ProcSleep() case and I start to have concern about the others WaitLatch()
that would/could be "woken up" every 1s.
Using disable_timeout() and enable_timeout_after() in WaitEventSetWait() does not
look like a great answer to this concern, so I wonder if we should use a larger
flush frequency instead (as proposed up-thread), thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 063826ae576..7376dd1f316 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -1322,6 +1322,7 @@ ProcSleep(LOCALLOCK *locallock)
bool allow_autovacuum_cancel = true;
bool logged_recovery_conflict = false;
ProcWaitStatus myWaitStatus;
+ bool anytime_timeout_was_active = false;
/* The caller must've armed the on-error cleanup mechanism */
Assert(GetAwaitedLock() == locallock);
@@ -1398,6 +1399,10 @@ ProcSleep(LOCALLOCK *locallock)
standbyWaitStart = GetCurrentTimestamp();
}
+ anytime_timeout_was_active = get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT);
+ if (anytime_timeout_was_active)
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+
/*
* If somebody wakes us between LWLockRelease and WaitLatch, the latch
* will not wait. But a set latch does not necessarily mean that the lock
@@ -1661,6 +1666,9 @@ ProcSleep(LOCALLOCK *locallock)
}
} while (myWaitStatus == PROC_WAIT_STATUS_WAITING);
+ if (anytime_timeout_was_active)
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
/*
* Disable the timers, if they are still running. As in LockErrorCleanup,
* we must preserve the LOCK_TIMEOUT indicator flag: if a lock timeout has
Attachments:
[text/plain] fix_ProcSleep.txt (1.3K, 2-fix_ProcSleep.txt)
download | inline diff:
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 063826ae576..7376dd1f316 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -1322,6 +1322,7 @@ ProcSleep(LOCALLOCK *locallock)
bool allow_autovacuum_cancel = true;
bool logged_recovery_conflict = false;
ProcWaitStatus myWaitStatus;
+ bool anytime_timeout_was_active = false;
/* The caller must've armed the on-error cleanup mechanism */
Assert(GetAwaitedLock() == locallock);
@@ -1398,6 +1399,10 @@ ProcSleep(LOCALLOCK *locallock)
standbyWaitStart = GetCurrentTimestamp();
}
+ anytime_timeout_was_active = get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT);
+ if (anytime_timeout_was_active)
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+
/*
* If somebody wakes us between LWLockRelease and WaitLatch, the latch
* will not wait. But a set latch does not necessarily mean that the lock
@@ -1661,6 +1666,9 @@ ProcSleep(LOCALLOCK *locallock)
}
} while (myWaitStatus == PROC_WAIT_STATUS_WAITING);
+ if (anytime_timeout_was_active)
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
/*
* Disable the timers, if they are still running. As in LockErrorCleanup,
* we must preserve the LOCK_TIMEOUT indicator flag: if a lock timeout has
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-01-28 06:35 ` Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Michael Paquier @ 2026-01-28 06:35 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Fujii Masao <[email protected]>; Sami Imseih <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
On Mon, Jan 26, 2026 at 06:59:28AM +0000, Bertrand Drouvot wrote:
> The attached, to apply on top of 0001, fix the issue. However it handles only the
> WaitLatch in ProcSleep() case and I start to have concern about the others WaitLatch()
> that would/could be "woken up" every 1s.
Hmm, I indeed suspect that is may not be the only one.. This is much bigger.
> Using disable_timeout() and enable_timeout_after() in WaitEventSetWait() does not
> look like a great answer to this concern, so I wonder if we should use a larger
> flush frequency instead (as proposed up-thread), thoughts?
Only a larger frequency is not the correct answer here. It would just
reduce the frequency of the extra lock wait messages for one: these
should never appear more than necessary. And how about for example
extension code?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, 2-signature.asc)
download
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
@ 2026-01-28 11:14 ` Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Bertrand Drouvot @ 2026-01-28 11:14 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Fujii Masao <[email protected]>; Sami Imseih <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Wed, Jan 28, 2026 at 03:35:21PM +0900, Michael Paquier wrote:
> On Mon, Jan 26, 2026 at 06:59:28AM +0000, Bertrand Drouvot wrote:
> > The attached, to apply on top of 0001, fix the issue. However it handles only the
> > WaitLatch in ProcSleep() case and I start to have concern about the others WaitLatch()
> > that would/could be "woken up" every 1s.
>
> > Using disable_timeout() and enable_timeout_after() in WaitEventSetWait() does not
> > look like a great answer to this concern, so I wonder if we should use a larger
> > flush frequency instead (as proposed up-thread), thoughts?
>
> Only a larger frequency is not the correct answer here. It would just
> reduce the frequency of the extra lock wait messages for one: these
> should never appear more than necessary.
Right. The fix in fix_ProcSleep.txt shared up-thread solves that and has been
added in 0001 attached.
Also the attached is now split in 4 sub-patches with 0002 introducing a new
GUC to control the flush interval (default is 10s). Note that 0001 to 0003 could
be merged as one patch but I did it that way to ease the review.
The new version also adds more documentation and takes care of Sami's comments
shared up-thread.
> And how about for example extension code?
I think that depending on how they write their code around WaitLatch (if any) they
could see messages being reported (if they do) in the logs at stats_flush_interval
frequency.
That said the default value is 10s and that looks pretty long for a latch to wait on.
In any case, they should already take care of a latch being woken by WL_LATCH_SET.
What do you think?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-01-31 01:33 ` Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Sami Imseih @ 2026-01-31 01:33 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
> Also the attached is now split in 4 sub-patches with 0002 introducing a new
> GUC to control the flush interval (default is 10s). Note that 0001 to 0003 could
> be merged as one patch but I did it that way to ease the review.
Thanks for the patches!
I do like the GUC introduced in 0002 to control the ANYTIME stats
flush interval.
But, as I was looking at this a bit more today and testing it, I am
not quite sure
I like what is happening here:
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /*
+ * Exit if no pending stats at all. This avoids unnecessary work when
+ * backends are idle or in sessions without stats accumulation.
+ *
+ * Note: This check isn't precise as there might be only transactional
+ * stats pending, which we'll skip during the flush. However,
maintaining
+ * precise tracking would add complexity that does not seem
worth it from
+ * a performance point of view (no noticeable performance regression has
+ * been observed with the current implementation).
+ */
+ if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed)
+ return;
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
There is a check if pgStatPending is empty so we can return early.
However, with FLUSH_MIXED, the scenario is we will more than likely always
have TXN_BOUNDARY stats that can only be flushed at the end of a transaction.
This means that most of the time we will call pgstat_flush_pending_entries
and then leave it up to the anytime_cb to check for pending ANYTIME stats to
flush (before taking the lock ).
+bool
+pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ Oid dboid;
+ PgStat_TableStatus *lstats; /* pending stats entry */
+ PgStatShared_Relation *shtabstats;
+ PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
+ PgStat_StatDBEntry *dbentry; /* pending database entry */
+
+ dboid = entry_ref->shared_entry->key.dboid;
+ lstats = (PgStat_TableStatus *) entry_ref->pending;
+ shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (!(lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0))
+ return true;
This makes things confusing because instead of just relying on
dlist_is_empty(&pgStatPending) to check if we need to flush anything,
the responsibility now moves to the callback, which now also has to
account for all the ANYTIME fields.
The way I can think about making this better is to somehow track if
we have ANYTIME data that needs to be flushed a different way
( maybe a second pgStatPending list for anytime stats ?? )
What do you think?
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
@ 2026-01-31 17:16 ` Sami Imseih <[email protected]>
2026-02-02 07:31 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
0 siblings, 2 replies; 27+ messages in thread
From: Sami Imseih @ 2026-01-31 17:16 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
> This makes things confusing because instead of just relying on
> dlist_is_empty(&pgStatPending) to check if we need to flush anything,
> the responsibility now moves to the callback, which now also has to
> account for all the ANYTIME fields.
>
> The way I can think about making this better is to somehow track if
> we have ANYTIME data that needs to be flushed a different way
> ( maybe a second pgStatPending list for anytime stats ?? )
Thinking about this a bit more, it seems the FLUSH_MIXED may
not be needed. Instead can we just introduce a new global variable
called pgstat_report_variable_anytime which acts like
pgstat_report_fixed, except it's set to true whenever we update
anytime variable-numbered stats
```
#define pgstat_count_heap_scan(rel)
\
do {
\
pgstat_report_variable_anytime = true;
\
if (pgstat_should_count_relation(rel))
\
(rel)->pgstat_info->counts.numscans++;
\
} while (0)
````
and reset whenever we call pgstat_report_anytime_stat, like this:
```
void
pgstat_report_anytime_stat(bool force)
{
.....
.........
if (!pgstat_report_variable_anytime && !pgstat_report_fixed)
return;
/* Flush stats outside of transaction boundary */
if (pgstat_report_variable_anytime)
pgstat_flush_pending_entries(nowait, true);
....
........
pgstat_report_variable_anytime = false;
}
```
and then inside pgstat_flush_pending_entries, we just look
for kind_info->flush_mode == FLUSH_ANYTIME to flush.
```
/* flush the stats (with the appropriate callback), if possible */
if (anytime_only &&
kind_info->flush_mode == FLUSH_ANYTIME &&
kind_info->flush_anytime_cb != NULL)
{
/* Partial flush of non-transactional fields only */
did_flush = kind_info->flush_anytime_cb(entry_ref, nowait);
is_partial_flush = true;
}
````
With this approach, we will not enter pgstat_flush_pending_entries
inside pgstat_report_anytime_stat unless we have anytime
variable stats to report.
Also, the anytime flush callback does not need to check if there are
any variable-numbered stats to flush. This will not be needed as
it is in v4-0004
```
+ /*
+ * Check if there are any non-transactional stats to flush. Avoid
+ * unnecessarily locking the entry if nothing accumulated.
+ */
+ if (!(lstats->counts.numscans > 0 ||
+ lstats->counts.tuples_returned > 0 ||
+ lstats->counts.tuples_fetched > 0 ||
+ lstats->counts.blocks_fetched > 0 ||
+ lstats->counts.blocks_hit > 0))
+ return true;
```
This feels like an easier approach to reason about and we don't
need to add a third flush mode.
Thoughts?
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
@ 2026-02-02 07:31 ` Michael Paquier <[email protected]>
2026-02-02 17:19 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
1 sibling, 1 reply; 27+ messages in thread
From: Michael Paquier @ 2026-02-02 07:31 UTC (permalink / raw)
To: Sami Imseih <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
On Sat, Jan 31, 2026 at 11:16:03AM -0600, Sami Imseih wrote:
> Also, the anytime flush callback does not need to check if there are
> any variable-numbered stats to flush. This will not be needed as
> it is in v4-0004
>
> ```
> + /*
> + * Check if there are any non-transactional stats to flush. Avoid
> + * unnecessarily locking the entry if nothing accumulated.
> + */
> + if (!(lstats->counts.numscans > 0 ||
> + lstats->counts.tuples_returned > 0 ||
> + lstats->counts.tuples_fetched > 0 ||
> + lstats->counts.blocks_fetched > 0 ||
> + lstats->counts.blocks_hit > 0))
> + return true;
> ```
I think that this kind of check may finish by being expensive.
Reducing the number of fields to check for a non-flushing state, if
required, would be preferrable.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, 2-signature.asc)
download
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-02 07:31 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
@ 2026-02-02 17:19 ` Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Bertrand Drouvot @ 2026-02-02 17:19 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Sami Imseih <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Mon, Feb 02, 2026 at 04:31:48PM +0900, Michael Paquier wrote:
> On Sat, Jan 31, 2026 at 11:16:03AM -0600, Sami Imseih wrote:
> > Also, the anytime flush callback does not need to check if there are
> > any variable-numbered stats to flush. This will not be needed as
> > it is in v4-0004
> >
> > ```
> > + /*
> > + * Check if there are any non-transactional stats to flush. Avoid
> > + * unnecessarily locking the entry if nothing accumulated.
> > + */
> > + if (!(lstats->counts.numscans > 0 ||
> > + lstats->counts.tuples_returned > 0 ||
> > + lstats->counts.tuples_fetched > 0 ||
> > + lstats->counts.blocks_fetched > 0 ||
> > + lstats->counts.blocks_hit > 0))
> > + return true;
> > ```
>
> I think that this kind of check may finish by being expensive.
> Reducing the number of fields to check for a non-flushing state, if
> required, would be preferrable.
Done, in v5 just shared up-thread.
Oh and I forgot to mention that with the new design in v5, then the fix for
the issue discovered by Masao-san is not needed anymore (and so has been removed
from 0001).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
@ 2026-02-02 17:16 ` Bertrand Drouvot <[email protected]>
2026-02-04 15:57 ` Re: Flush some statistics within running transactions Zsolt Parragi <[email protected]>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
1 sibling, 2 replies; 27+ messages in thread
From: Bertrand Drouvot @ 2026-02-02 17:16 UTC (permalink / raw)
To: Sami Imseih <[email protected]>; +Cc: Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Sat, Jan 31, 2026 at 11:16:03AM -0600, Sami Imseih wrote:
> > This makes things confusing because instead of just relying on
> > dlist_is_empty(&pgStatPending) to check if we need to flush anything,
> > the responsibility now moves to the callback, which now also has to
> > account for all the ANYTIME fields.
> >
>
> Thinking about this a bit more, it seems the FLUSH_MIXED may
> not be needed. Instead can we just introduce a new global variable
> called pgstat_report_variable_anytime which acts like
> pgstat_report_fixed, except it's set to true whenever we update
> anytime variable-numbered stats
>
> With this approach, we will not enter pgstat_flush_pending_entries
> inside pgstat_report_anytime_stat unless we have anytime
> variable stats to report.
Thanks for looking at it!
In v5 attached I changed the design so that we don't re-enable the timeout
after each stats flush in ProcessInterrupts(). Instead the timeout is enabled
when we set pgstat_report_fixed to true or in pgstat_prep_pending_entry() when
appropriate. So that we know that when we enter pgstat_report_anytime_stat() that's
for good reasons and we don't need extra checks in it.
> ```
> + /*
> + * Check if there are any non-transactional stats to flush. Avoid
> + * unnecessarily locking the entry if nothing accumulated.
> + */
> + if (!(lstats->counts.numscans > 0 ||
> + lstats->counts.tuples_returned > 0 ||
> + lstats->counts.tuples_fetched > 0 ||
> + lstats->counts.blocks_fetched > 0 ||
> + lstats->counts.blocks_hit > 0))
> + return true;
> ```
Yeah, with the new design in place then those are not needed anymore.
> This feels like an easier approach to reason about and we don't
> need to add a third flush mode.
I do think we still need it. Indeed in 0004 that helps distinguish between
anytime flush or mixed flush (with the help of the new pgstat_report_mixed_anytime
global variable) in pgstat_prep_pending_entry().
Thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v5-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (28.5K, 2-v5-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From bddc98bb813e2a0453112b47431d324657eabaab Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v5 1/4] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling
pgstat_report_anytime_stat(false)
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 9 ++
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 118 ++++++++++++++++----
src/backend/utils/activity/pgstat_backend.c | 9 ++
src/backend/utils/activity/pgstat_io.c | 5 +
src/backend/utils/activity/pgstat_slru.c | 5 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 4 +
src/include/utils/pgstat_internal.h | 21 ++++
src/include/utils/timeout.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
19 files changed, 213 insertions(+), 25 deletions(-)
5.5% src/backend/access/transam/
15.3% src/backend/postmaster/
3.4% src/backend/replication/
3.8% src/backend/tcop/
57.0% src/backend/utils/activity/
9.2% src/include/utils/
5.3% src/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..9503aea5b4d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,10 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2070,11 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
+ PGSTAT_MIN_INTERVAL);
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 80e3088fc7e..ab5d0645026 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -104,7 +106,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -114,6 +116,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 6482c21b8f9..6e187315613 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -216,7 +217,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -226,6 +227,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index a1a4f65f9a9..498d147f0da 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -246,6 +247,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c3d56c866d3..cec5dfdb430 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -250,7 +252,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -272,6 +274,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 38ec8a4c8c7..7416ca703c9 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -107,7 +109,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -117,6 +119,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 8b99160ed0e..24d7ef795cb 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -253,7 +255,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -261,6 +263,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index b4a8d2f3a1c..d19aa45400d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3530,6 +3530,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..2c9454677e9 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -112,6 +112,7 @@
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +123,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -388,6 +394,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
.shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
@@ -404,6 +411,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter),
.shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
@@ -420,6 +428,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer),
.shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
@@ -436,6 +445,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +463,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +481,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +787,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,12 +1293,18 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
entry_ref->pending = MemoryContextAllocZero(pgStatPendingContext, entrysize);
dlist_push_tail(&pgStatPending, &entry_ref->pending_node);
+
+ /* Schedule next anytime stats update timeout */
+ if (kind_info->flush_mode == FLUSH_ANYTIME && IsUnderPostmaster &&
+ !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
}
return entry_ref;
@@ -1345,9 +1351,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,6 +1388,20 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
@@ -1402,6 +1427,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2171,31 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
+
+/*
+ * Timeout handler for flushing non-transactional stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 1350f5f62f1..9dcb24db975 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,10 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +87,10 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..53dbf2a514b 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,10 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
have_iostats = true;
pgstat_report_fixed = true;
}
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..1d16cde1889 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -223,6 +224,10 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..f45365f47f7 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -765,6 +766,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..8aeb9628871 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..1651f16f966 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,6 +536,7 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
extern void pgstat_reset_counters(void);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..a9190078d0e 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,13 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * Some stats have to be updated only at transaction boundaries (such as
+ * tuples_inserted updated, deleted), so it's very important to set the
+ * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default).
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -677,6 +697,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9f5ee8fd482..860f835c088 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v5-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch (10.5K, 3-v5-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch)
download | inline diff:
From c64a3a582201fc7d1925d374ad6fa5778b0d4d14 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v5 2/4] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 32 +++++++++++++++++++
src/backend/access/transam/xlog.c | 4 +--
src/backend/utils/activity/pgstat.c | 16 +++++++++-
src/backend/utils/activity/pgstat_backend.c | 4 +--
src/backend/utils/activity/pgstat_io.c | 2 +-
src/backend/utils/activity/pgstat_slru.c | 2 +-
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/pgstat.h | 1 +
src/include/utils/guc_hooks.h | 1 +
10 files changed, 66 insertions(+), 7 deletions(-)
45.9% doc/src/sgml/
6.5% src/backend/access/transam/
31.8% src/backend/utils/activity/
12.3% src/backend/utils/misc/
3.2% src/include/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5560b95ee60..3136816a933 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8834,6 +8834,38 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which non-transactional statistics are made visible
+ during running transactions. Non-transactional statistics include, for
+ example, WAL activity and I/O operations.
+ They become visible at that interval in monitoring views such as
+ <link linkend="monitoring-pg-stat-io-view"> <structname>pg_stat_io</structname></link>
+ and <link linkend="monitoring-pg-stat-wal-view"> <structname>pg_stat_wal</structname></link>
+ during running transactions.
+ If this value is specified without units, it is taken as milliseconds.
+ The default is 10 seconds (<literal>10s</literal>), which is probably
+ about the smallest value you would want in practice for long running
+ transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect transactional statistics such as
+ <structname>pg_stat_all_tables</structname> columns (like
+ <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ <structfield>n_tup_del</structfield>), which are always flushed at transaction
+ boundaries to maintain consistency.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 9503aea5b4d..31523dea923 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1087,7 +1087,7 @@ XLogInsertRecord(XLogRecData *rdata,
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
@@ -2073,7 +2073,7 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
- PGSTAT_MIN_INTERVAL);
+ pgstat_flush_interval);
/*
* Required for the flush of pending stats WAL data, per
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 2c9454677e9..dd174129403 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -203,6 +203,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -1304,7 +1305,7 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
/* Schedule next anytime stats update timeout */
if (kind_info->flush_mode == FLUSH_ANYTIME && IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
}
return entry_ref;
@@ -2172,6 +2173,19 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ {
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+ }
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 9dcb24db975..f5b8c7b039c 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -69,7 +69,7 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
backend_has_iostats = true;
pgstat_report_fixed = true;
@@ -89,7 +89,7 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
backend_has_iostats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 53dbf2a514b..b69a1e26f7d 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -82,7 +82,7 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
have_iostats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 1d16cde1889..36231ee874b 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -226,7 +226,7 @@ get_slru_entry(int slru_idx)
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index f0260e6e412..3bb43362e51 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2782,6 +2782,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c4f92fcdac8..6ce5a250170 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -669,6 +669,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 1651f16f966..e0f222695bf 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -816,6 +816,7 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index b6ecb0e769f..3a2ae6c41cd 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
--
2.34.1
[text/x-diff] v5-0003-Remove-useless-calls-to-flush-some-stats.patch (7.6K, 4-v5-0003-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From 73aecb37f8d6f9551085a91bc731298d753d6525 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v5 3/4] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.9% src/backend/replication/
22.8% src/backend/utils/activity/
3.5% src/test/recovery/t/
3.6% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 24d7ef795cb..1c5ffcab3e0 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -572,16 +572,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a0e6a3d200c..74102def9c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1825,7 +1821,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1846,7 +1841,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1957,8 +1951,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1967,15 +1960,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2878,8 +2862,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2974,9 +2956,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2984,7 +2963,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -2995,21 +2973,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..feae2ae5f44 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..c058a5f9b1f 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index d7e62e4d488..dda872f7074 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v5-0004-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch (20.9K, 5-v5-0004-Add-FLUSH_MIXED-support-and-implement-it-for-RELA.patch)
download | inline diff:
From 16df908878b6ab3e3295785e0298b024da457ec7 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v5 4/4] Add FLUSH_MIXED support and implement it for RELATION
stats
This commit extends the non transactional stats infrastructure to support statistics
kinds with mixed transaction behavior: some fields are transactional (e.g., tuple
inserts/updates/deletes) while others are non transactional (e.g., sequential scans
blocks read, ...).
It introduces FLUSH_MIXED as a third flush mode type, alongside FLUSH_ANYTIME
and FLUSH_AT_TXN_BOUNDARY. For FLUSH_MIXED kinds, a new flush_anytime_cb callback
enables partial flushing of only the non transactional fields during running
transactions.
Some tests are also added.
Implementation details:
- Add FLUSH_MIXED to PgStat_FlushMode enum
- Add flush_anytime_cb to PgStat_KindInfo for partial flushing callback
- Update pgstat_flush_pending_entries() to call flush_anytime_cb for
FLUSH_MIXED entries when in anytime_only mode
- Keep FLUSH_MIXED entries in the pending list after partial flush, as
transactional fields still need to be flushed at transaction boundary
- Add pgstat_report_mixed_anytime, a new global variable to track when the
ANYTIME_STATS_UPDATE_TIMEOUT needs to be enabled for FLUSH_MIXED mode
RELATION stats are making use of FLUSH_MIXED:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_MIXED
- Implement pgstat_relation_flush_anytime_cb() to flush only read related
stats: numscans, tuples_returned, tuples_fetched, blocks_fetched,
blocks_hit
- Clear these fields after flushing to prevent double counting when
pgstat_relation_flush_cb() runs at transaction commit
- Transactional stats (tuples_inserted, tuples_updated, tuples_deleted,
live_tuples, dead_tuples) remain pending until transaction boundary
The DATABASE kind is also changed from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME, so
that some stats inherited from relations stats are also visible while the transaction
is in progress.
Remark:
We could also imagine adding a new flush_anytime_static_cb() callback for
future FLUSH_MIXED fixed amount stats.
---
doc/src/sgml/monitoring.sgml | 29 +++++++++
src/backend/utils/activity/pgstat.c | 44 ++++++++++---
src/backend/utils/activity/pgstat_relation.c | 67 ++++++++++++++++++++
src/include/pgstat.h | 10 +++
src/include/utils/pgstat_internal.h | 11 ++++
src/test/isolation/expected/stats.out | 40 ++++++++++++
src/test/isolation/expected/stats_1.out | 40 ++++++++++++
src/test/isolation/specs/stats.spec | 17 ++++-
8 files changed, 247 insertions(+), 11 deletions(-)
14.3% doc/src/sgml/
43.1% src/backend/utils/activity/
4.9% src/include/utils/
4.6% src/include/
27.5% src/test/isolation/expected/
5.3% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..581d6ea7811 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index dd174129403..d54ccb95c6a 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -219,6 +219,12 @@ PgStat_LocalState pgStatLocal;
*/
bool pgstat_report_fixed = false;
+/*
+ * Track pending reports for mixed anytime stats, used by
+ * pgstat_report_anytime_stat().
+ */
+bool pgstat_report_mixed_anytime = false;
+
/* ----------
* Local data
*
@@ -289,7 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -307,7 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_MIXED,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -315,6 +321,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.pending_size = sizeof(PgStat_TableStatus),
.flush_pending_cb = pgstat_relation_flush_cb,
+ .flush_anytime_cb = pgstat_relation_flush_anytime_cb,
.delete_pending_cb = pgstat_relation_delete_pending_cb,
.reset_timestamp_cb = pgstat_relation_reset_timestamp_cb,
},
@@ -1303,8 +1310,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
dlist_push_tail(&pgStatPending, &entry_ref->pending_node);
/* Schedule next anytime stats update timeout */
- if (kind_info->flush_mode == FLUSH_ANYTIME && IsUnderPostmaster &&
- !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ if ((kind_info->flush_mode == FLUSH_ANYTIME || pgstat_report_mixed_anytime) &&
+ IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval);
}
@@ -1353,10 +1360,11 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME and FLUSH_MIXED entries,
+ * using flush_anytime_cb for FLUSH_MIXED.
* This is safe to call inside transactions.
*
- * If anytime_only is false, flushes all entries.
+ * If anytime_only is false, flushes all entries using flush_pending_cb.
*/
static bool
pgstat_flush_pending_entries(bool nowait, bool anytime_only)
@@ -1384,6 +1392,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1403,8 +1412,21 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
continue;
}
- /* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ /* flush the stats (with the appropriate callback), if possible */
+ if (anytime_only &&
+ kind_info->flush_mode == FLUSH_MIXED &&
+ kind_info->flush_anytime_cb != NULL)
+ {
+ /* Partial flush of non-transactional fields only */
+ did_flush = kind_info->flush_anytime_cb(entry_ref, nowait);
+ is_partial_flush = true;
+ }
+ else
+ {
+ /* Full flush */
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ is_partial_flush = false;
+ }
Assert(did_flush || nowait);
@@ -1414,8 +1436,8 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
@@ -2201,6 +2223,8 @@ pgstat_report_anytime_stat(bool force)
/* Flush stats outside of transaction boundary */
pgstat_flush_pending_entries(nowait, true);
pgstat_flush_fixed_stats(nowait, true);
+
+ pgstat_report_mixed_anytime = false;
}
/*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index feae2ae5f44..ae8778d8e39 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -887,6 +887,73 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+/*
+ * Flush only non-transactional relation stats.
+ *
+ * This is called periodically during running transactions to make some
+ * statistics visible without waiting for the transaction to finish.
+ *
+ * Transactional stats (inserts/updates/deletes and their effects on live/dead
+ * tuple counts) remain in pending until the transaction ends, at which point
+ * pgstat_relation_flush_cb() will flush them.
+ *
+ * If nowait is true and the lock could not be immediately acquired, returns
+ * false without flushing the entry. Otherwise returns true.
+ */
+bool
+pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ Oid dboid;
+ PgStat_TableStatus *lstats; /* pending stats entry */
+ PgStatShared_Relation *shtabstats;
+ PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
+ PgStat_StatDBEntry *dbentry; /* pending database entry */
+
+ dboid = entry_ref->shared_entry->key.dboid;
+ lstats = (PgStat_TableStatus *) entry_ref->pending;
+ shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ /* Add only the non-transactional values to the shared entry */
+ tabentry = &shtabstats->stats;
+
+ tabentry->numscans += lstats->counts.numscans;
+ if (lstats->counts.numscans)
+ {
+ TimestampTz t = GetCurrentTimestamp();
+
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+ tabentry->tuples_returned += lstats->counts.tuples_returned;
+ tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+ tabentry->blocks_fetched += lstats->counts.blocks_fetched;
+ tabentry->blocks_hit += lstats->counts.blocks_hit;
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ dbentry->tuples_returned += lstats->counts.tuples_returned;
+ dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ dbentry->blocks_fetched += lstats->counts.blocks_fetched;
+ dbentry->blocks_hit += lstats->counts.blocks_hit;
+
+ /*
+ * Clear the flushed fields from pending stats to prevent double-counting
+ * when pgstat_relation_flush_cb() runs at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
+}
+
void
pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref)
{
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e0f222695bf..66cc7745498 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -38,6 +38,9 @@
/* Minimum interval non-forced stats flushes */
#define PGSTAT_MIN_INTERVAL 1000
+/* Track if mixed anytime stats need to be flushed */
+extern PGDLLIMPORT bool pgstat_report_mixed_anytime;
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -693,36 +696,43 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.numscans++; \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.tuples_returned++; \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.tuples_fetched++; \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.numscans++; \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.tuples_returned += (n); \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.blocks_fetched++; \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.blocks_hit++; \
} while (0)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index a9190078d0e..db4d86cf31c 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -236,6 +236,8 @@ typedef enum PgStat_FlushMode
* transaction boundary */
FLUSH_ANYTIME, /* All fields can be flushed anytime,
* including within transactions */
+ FLUSH_MIXED, /* Mix of fields that can be flushed anytime
+ * or only at transaction boundary */
} PgStat_FlushMode;
/*
@@ -271,6 +273,12 @@ typedef struct PgStat_KindInfo
*/
PgStat_FlushMode flush_mode;
+ /*
+ * For FLUSH_MIXED kinds: callback to flush only some fields. If NULL for
+ * a MIXED kind, treated as FLUSH_AT_TXN_BOUNDARY.
+ */
+ bool (*flush_anytime_cb) (PgStat_EntryRef *entry_ref, bool nowait);
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -784,6 +792,7 @@ extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_anytime_cb(PgStat_EntryRef *entry_ref, bool nowait);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -883,6 +892,8 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 obji
*/
extern PGDLLIMPORT bool pgstat_report_fixed;
+/* Track if mixed anytime stats need to be flushed */
+
/* Backend-local stats state */
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..6d62b30e4a7 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..2fade10e817 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,46 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 0| 0| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..a4084efda49 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -435,6 +441,15 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
--
2.34.1
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-02-04 15:57 ` Zsolt Parragi <[email protected]>
2026-02-04 20:15 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
1 sibling, 1 reply; 27+ messages in thread
From: Zsolt Parragi @ 2026-02-04 15:57 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Sami Imseih <[email protected]>; Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]
Hello
do { \
+ pgstat_report_mixed_anytime = true; \
if (pgstat_should_count_relation(rel)) \
(rel)->pgstat_info->counts.numscans++; \
Shouldn't these pgstat_report_mixed_anytime changes go inside the if
statement in all macros?
+/* Track if mixed anytime stats need to be flushed */
+
/* Backend-local stats state */
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
This seems to be a leftover comment
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-02-04 15:57 ` Re: Flush some statistics within running transactions Zsolt Parragi <[email protected]>
@ 2026-02-04 20:15 ` Sami Imseih <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Sami Imseih @ 2026-02-04 20:15 UTC (permalink / raw)
To: Zsolt Parragi <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]
> do { \
> + pgstat_report_mixed_anytime = true; \
> if (pgstat_should_count_relation(rel)) \
> (rel)->pgstat_info->counts.numscans++; \
>
> Shouldn't these pgstat_report_mixed_anytime changes go inside the if
> statement in all macros?
I think you are correct here, and there is an even more fundamental issue.
since
``
#define pgstat_should_count_relation(rel) \
(likely((rel)->pgstat_info != NULL) ? true : \
```
could return if there is already a pgstat_info, we may never actually
enable the timeout.
so, I think we should:
1/
remove the scheduling of the timeout from pgstat_prep_pending_entry
@@ -1308,11 +1308,6 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid
dboid, uint64 objid, bool *creat
entry_ref->pending =
MemoryContextAllocZero(pgStatPendingContext, entrysize);
dlist_push_tail(&pgStatPending, &entry_ref->pending_node);
-
- /* Schedule next anytime stats update timeout */
- if ((kind_info->flush_mode == FLUSH_ANYTIME ||
pgstat_report_mixed_anytime) &&
- IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
-
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
pgstat_flush_interval);
}
2/
Create a routine to schedule the timeout:
+void
+ScheduleAnyTimeUpdate(void)
+{
+ /* Schedule next anytime stats update timeout */
+ if (IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
pgstat_flush_interval);
+
+ pgstat_report_mixed_anytime = true;
+}
This will also set pgstat_report_mixed_anytime to true. In my earlier
comments, I mentioned
having this as a public routine will be needed for extensions that
register a custom kind as well.
3/
All the count routines that wish to schedule any ANYTIME update because
their kind allows it can do so with ScheduleAnyTimeUpdate(). In the relation
stats, this can happen after it is checked that the relation should
track counts.
+#define pgstat_count_heap_scan(rel) \
+ do { \
+ if (pgstat_should_count_relation(rel)) \
+ { \
+ (rel)->pgstat_info->counts.numscans++; \
+ ScheduleAnyTimeUpdate(); \
+ } \
+ } while (0)
Also, it would be good to check if we have anytime flushes of either a mixed or
a fixed kind. Not strictly necessary, but I think it's better to avoid
needlessly entering
the flush routines.
@@ -2220,11 +2215,27 @@ pgstat_report_anytime_stat(bool force)
pgstat_assert_is_up();
/* Flush stats outside of transaction boundary */
- pgstat_flush_pending_entries(nowait, true);
- pgstat_flush_fixed_stats(nowait, true);
+ if (pgstat_report_mixed_anytime)
+ pgstat_flush_pending_entries(nowait, true);
+
+ if (pgstat_report_mixed_anytime)
+ pgstat_flush_fixed_stats(nowait, true);
pgstat_report_mixed_anytime = false;
+ pgstat_report_fixed = false;
}
I think we could benefit from a test that st track_counts to OFF inside
a transaction and re-enables it, to make sure the pgstat_should_count_relation
work is done correctly.
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-02-04 16:19 ` Sami Imseih <[email protected]>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
1 sibling, 1 reply; 27+ messages in thread
From: Sami Imseih @ 2026-02-04 16:19 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
> In v5 attached I changed the design so that we don't re-enable the timeout
Thanks!
> I do think we still need it. Indeed in 0004 that helps distinguish between
> anytime flush or mixed flush (with the help of the new pgstat_report_mixed_anytime
> global variable) in pgstat_prep_pending_entry().
Will address below in the comments.
v5-0001:
1/
```
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
```
FLUSH_ANYTIME does not have any impact on Archiver. It does not have
a .flush_static_cb registered and entries are flushed directly to
shared memory via `pgstat_report_archiver` inside the `pgarch_ArchiverCopyLoop`
loop.
FLUSH_ANYTIME needs to be described a bit more that it only applies to
kinds that have flush callbacks.
2/
I suggest simplifying the code comment to this:
```
/*
- * Some stats have to be updated only at transaction boundaries (such as
- * tuples_inserted updated, deleted), so it's very important to set the
- * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default).
+ * The mode of when to flush stats. See PgStat_FlushMode for
more details.
*/
PgStat_FlushMode flush_mode;
```
PgStat_FlushMode has sufficient description.
3/
we have this pattern:
```
/* Schedule next anytime stats update timeout */
if (IsUnderPostmaster &&
!get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT,
pgstat_flush_interval);
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
```
to report fixed stats. I think it will be good to turn this into a public
routine, i.e. pgstat_report_fixed_entry, that can also be used by extensions
that register custom kinds with ANYTIME flush.
extensions using variable-numbered will get the timeout enabled
via pgstat_prep_pending_entry, per v5-0004.
We may want to add a test in test_custom_stats to ensure
this ANYTIME mechanism can work with custom kinds. What do you think?
v5-0002:
Overall, this GUC seems like a good idea. 10 second default and 1
second minimum are fine.
The patch overall looks solid, and I could not find any issues.
I do have a suggestion about the documentation:
```
</term>
<listitem>
<para>
- Sets the interval at which non-transactional statistics are
made visible
- during running transactions. Non-transactional statistics include, for
- example, WAL activity and I/O operations.
- They become visible at that interval in monitoring views such as
- <link linkend="monitoring-pg-stat-io-view">
<structname>pg_stat_io</structname></link>
- and <link linkend="monitoring-pg-stat-wal-view">
<structname>pg_stat_wal</structname></link>
- during running transactions.
- If this value is specified without units, it is taken as milliseconds.
- The default is 10 seconds (<literal>10s</literal>), which is probably
- about the smallest value you would want in practice for long running
- transactions.
+ Sets the interval at which statistics that can be updated
while a transaction is still running
+ are made visible. These include, for example, WAL activity
and I/O operations.
+ Such statistics are refreshed at the specified interval and
can be observed during active
+ transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>
and
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>.
+ Other statistics are only made visible at transaction end and
are not affected by this setting.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is
generally the smallest practical
+ value for long-running transactions.
</para>
<note>
- <para>
- This parameter does not affect transactional statistics such as
- <structname>pg_stat_all_tables</structname> columns (like
- <structfield>n_tup_ins</structfield>,
<structfield>n_tup_upd</structfield>,
- <structfield>n_tup_del</structfield>), which are always
flushed at transaction
- boundaries to maintain consistency.
- </para>
+ <para> This parameter does not affect statistics that are
only reported at transaction end,
+ such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>,
<structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics
are always flushed at the end of
+ a transaction. </para>
</note>
</listitem>
</varlistentry>
```
specifically, I want to avoid using "non-transactional", "transaction
boundary" terms, as
they may be confusing.
v5-0003: looks straightforward. I have no comments.
v5-0004:
/1
+ FLUSH_MIXED, /* Mix of fields that
can be flushed anytime
+ * or
only at transaction boundary */
I still don't think this is needed. I fail to see what value it adds.
When we set pgstat_report_mixed_anytime inside the pgstat_count functions,
which make it clear which of the fields are anytime fields.
2/
This is an oversight comment, it seems.
```
extern PGDLLIMPORT bool pgstat_report_fixed;
+/* Track if mixed anytime stats need to be flushed */
+
/* Backend-local stats state */
extern PGDLLIMPORT PgStat_LocalState pgStatLocal;
```
3/
```
+ tabentry->numscans += lstats->counts.numscans;
+ if (lstats->counts.numscans)
+ {
+ TimestampTz t = GetCurrentTimestamp();
```
Considering the 10 second default, GetCurrentTimestamp() should not be
an issue here.
The doc does need to be updated. From:
"The time of the last sequential scan on this table, based on the most
recent transaction stop time"
To this?
"The approximate time of the last sequential scan on this table,
updated at least every STATS_FLUSH_INTERVAL"
same for `last_index_scan`
4/
nit: /non partial flush/non-partial flush/
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
@ 2026-02-04 16:26 ` Sami Imseih <[email protected]>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Sami Imseih @ 2026-02-04 16:26 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
> Will address below in the comments.
Sorry there was one more comment I forgot about.
in v5-0004, instead of `flush_anytime_cb`, can't we
just pass the anytime flag to the `flush_pending_cb` and
take care of what needs to be flushed there?
It will be up to the author of the kind if they want to further
split work in multiple functions, but we could just have a
single callback that is ANYTIME aware.
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
@ 2026-02-06 11:09 ` Bertrand Drouvot <[email protected]>
2026-02-14 02:23 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Bertrand Drouvot @ 2026-02-06 11:09 UTC (permalink / raw)
To: Sami Imseih <[email protected]>; +Cc: Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Wed, Feb 04, 2026 at 10:26:35AM -0600, Sami Imseih wrote:
> > Will address below in the comments.
>
> Sorry there was one more comment I forgot about.
>
> in v5-0004, instead of `flush_anytime_cb`, can't we
> just pass the anytime flag to the `flush_pending_cb` and
> take care of what needs to be flushed there?
>
> It will be up to the author of the kind if they want to further
> split work in multiple functions, but we could just have a
> single callback that is ANYTIME aware.
Thank you Sami and Zsolt for the reviews!
PFA attached v6, addressing the reviews comments.
Main changes are:
0001:
- Adds a pgstat_schedule_anytime_update() macro to schedule the next anytime flush
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducing
a new callback) in 0005.
0002:
Add some tests for anytime flush on custom stats. It adds a few functions:
we could/may want to avoid some duplicate code and update the version to 1.1
though. For the former, I don't have a strong opinion and for the later I'm not
sure (for example 07ff701dbd53 did not bump the version).
0005:
Get rid of the FLUSH_MIXED mode, the new anytime callback and the
pgstat_report_mixed_anytime variable introduced in v5. Instead v6:
- change RELATION and DATABASE kinds from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats
Remark:
I did not add any additional checks in pgstat_report_anytime_stat(), since we
enter it for good reason: at least one the flush has to be triggered. I'm not sure
it's worth to add more check to try to discard one of them. The flush functions
return early if they don't have anything to flush anyway. So the gain would be
to try to avoid one function call at pgstat_flush_interval interval, but we would
need to maitain the variable(s) and avoid possible race conditions with
pgstat_report_stat().
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[text/x-diff] v6-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch (41.9K, 2-v6-0001-Add-pgstat_report_anytime_stat-for-periodic-stats.patch)
download | inline diff:
From b2e59b407ca3fd60f5f917c9ec5f93047d0ac649 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Mon, 5 Jan 2026 09:41:39 +0000
Subject: [PATCH v6 1/5] Add pgstat_report_anytime_stat() for periodic stats
flushing
Long running transactions can accumulate significant statistics (WAL, IO, ...)
that remain unflushed until the transaction ends. This delays visibility of
resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces
spikes when flushed.
This commit introduces pgstat_report_anytime_stat(), which flushes
non transactional statistics even inside active transactions. A new timeout
handler fires every second (if enabled while adding pending stats) to call this
function, ensuring timely stats visibility without waiting for transaction completion.
Implementation details:
- Add PgStat_FlushMode enum to classify stats kinds:
* FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...)
* FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries
- Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats()
to accept a boolean anytime_only parameter:
* When false: flushes all stats (existing behavior)
* When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats
- The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only
boolean parameter. Most of the time it's not used (except for assertions), but it's
preparatory work for moving the relations stats to anytime (without introducin
a new callback).
- Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush,
relying on PGSTAT_MIN_INTERVAL
The force parameter in pgstat_report_anytime_stat() is currently unused (always
called with force=false) but reserved for future use cases requiring immediate
flushing.
---
src/backend/access/transam/xlog.c | 6 +
src/backend/postmaster/bgwriter.c | 9 +-
src/backend/postmaster/checkpointer.c | 10 +-
src/backend/postmaster/startup.c | 2 +
src/backend/postmaster/walsummarizer.c | 9 +-
src/backend/postmaster/walwriter.c | 9 +-
src/backend/replication/walreceiver.c | 9 +-
src/backend/replication/walsender.c | 8 +-
src/backend/tcop/postgres.c | 12 ++
src/backend/utils/activity/pgstat.c | 112 ++++++++++++++----
src/backend/utils/activity/pgstat_backend.c | 13 +-
src/backend/utils/activity/pgstat_bgwriter.c | 2 +-
.../utils/activity/pgstat_checkpointer.c | 2 +-
src/backend/utils/activity/pgstat_database.c | 2 +-
src/backend/utils/activity/pgstat_function.c | 4 +-
src/backend/utils/activity/pgstat_io.c | 10 +-
src/backend/utils/activity/pgstat_relation.c | 12 +-
src/backend/utils/activity/pgstat_slru.c | 6 +-
.../utils/activity/pgstat_subscription.c | 4 +-
src/backend/utils/activity/pgstat_wal.c | 10 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 3 +
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 16 +++
src/include/utils/pgstat_internal.h | 52 ++++++--
src/include/utils/timeout.h | 1 +
.../test_custom_stats/test_custom_var_stats.c | 4 +-
src/tools/pgindent/typedefs.list | 1 +
28 files changed, 264 insertions(+), 66 deletions(-)
10.8% src/backend/postmaster/
6.0% src/backend/replication/
50.7% src/backend/utils/activity/
6.0% src/backend/
19.3% src/include/utils/
5.6% src/include/
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..d01b11c7470 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata,
pgWalUsage.wal_fpi += num_fpi;
pgWalUsage.wal_fpi_bytes += fpi_bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/* Required for the flush of pending stats WAL data */
pgstat_report_fixed = true;
}
@@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic)
pgWalUsage.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
/*
* Required for the flush of pending stats WAL data, per
* update of pgWalUsage.
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0956bd39a85..059c601c3b8 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -49,7 +49,9 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
/*
@@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* We just started, assume there has been either a shutdown or
* end-of-recovery snapshot.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..e11c4b099c8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -66,8 +66,9 @@
#include "utils/acl.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
-
+#include "utils/timeout.h"
/*----------
* Shared memory area for communication between checkpointer and backends
@@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
@@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Initialize so that first time-driven event happens at the correct time.
*/
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..4954fe425b7 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -32,6 +32,7 @@
#include "storage/standby.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/timeout.h"
@@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler);
RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler);
RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
/*
* Unblock signals (they were blocked when the postmaster forked us)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 2d8f57099fd..9f8ef8159d1 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -48,6 +48,8 @@
#include "storage/shmem.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h"
/*
@@ -249,7 +251,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -271,6 +273,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* If an exception is encountered, processing resumes here.
*/
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 23e79a32345..ded0f250288 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -61,7 +61,9 @@
#include "storage/smgr.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/resowner.h"
+#include "utils/timeout.h"
/*
@@ -106,7 +108,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SignalHandlerForShutdownRequest);
pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN); /* not used */
@@ -116,6 +118,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
*/
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/*
* Create a memory context that we will do all our work in. We do this so
* that we can reset the context during error recovery and thereby avoid
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 10e64a7d1f4..11b7c114d3b 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -77,7 +77,9 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/pg_lsn.h"
+#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, die); /* request shutdown */
/* SIGQUIT handler was already set up by InitPostmasterChild */
- pqsignal(SIGALRM, SIG_IGN);
+ InitializeTimeouts(); /* establishes SIGALRM handler */
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SIG_IGN);
@@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
/* Reset some signals that are accepted by postmaster but not here */
pqsignal(SIGCHLD, SIG_DFL);
+ /*
+ * Register timeouts needed
+ */
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler);
+
/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);
if (WalReceiverFunctions == NULL)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2cde8ebc729..a7214d0dc6f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
@@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (TimestampDifferenceExceeds(last_flush, now,
WALSENDER_STATS_FLUSH_INTERVAL))
{
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
last_flush = now;
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 02e9aaa6bca..c7bc409b06f 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3509,6 +3509,18 @@ ProcessInterrupts(void)
pgstat_report_stat(true);
}
+ /*
+ * Flush stats outside of transaction boundary if the timeout fired.
+ * Unlike transactional stats, these can be flushed even inside a running
+ * transaction.
+ */
+ if (AnytimeStatsUpdateTimeoutPending)
+ {
+ AnytimeStatsUpdateTimeoutPending = false;
+
+ pgstat_report_anytime_stat(false);
+ }
+
if (ProcSignalBarrierPending)
ProcessProcSignalBarrier();
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 11bb71cad5a..411b65aae3e 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -112,6 +112,7 @@
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
#include "utils/timestamp.h"
@@ -122,8 +123,6 @@
* ----------
*/
-/* minimum interval non-forced stats flushes.*/
-#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void);
static void pgstat_reset_after_failure(void);
-static bool pgstat_flush_pending_entries(bool nowait);
+static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only);
+static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only);
static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
@@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.shared_size = sizeof(PgStatShared_Function),
.shared_data_off = offsetof(PgStatShared_Function, stats),
@@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
.accessed_across_databases = true,
@@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
+ .flush_mode = FLUSH_AT_TXN_BOUNDARY,
/* so pg_stat_subscription_stats entries can be seen in all databases */
.accessed_across_databases = true,
@@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = false,
+ .flush_mode = FLUSH_ANYTIME,
.accessed_across_databases = true,
@@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
@@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
@@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = true,
.write_to_file = true,
+ .flush_mode = FLUSH_ANYTIME,
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
@@ -775,23 +784,11 @@ pgstat_report_stat(bool force)
partial_flush = false;
/* flush of variable-numbered stats tracked in pending entries list */
- partial_flush |= pgstat_flush_pending_entries(nowait);
+ partial_flush |= pgstat_flush_pending_entries(nowait, false);
/* flush of other stats kinds */
if (pgstat_report_fixed)
- {
- for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
- {
- const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
-
- if (!kind_info)
- continue;
- if (!kind_info->flush_static_cb)
- continue;
-
- partial_flush |= kind_info->flush_static_cb(nowait);
- }
- }
+ partial_flush |= pgstat_flush_fixed_stats(nowait, false);
last_flush = now;
@@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
if (entry_ref->pending == NULL)
{
- size_t entrysize = pgstat_get_kind_info(kind)->pending_size;
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ size_t entrysize = kind_info->pending_size;
Assert(entrysize != (size_t) -1);
@@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
/*
* Flush out pending variable-numbered stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * This is safe to call inside transactions.
+ *
+ * If anytime_only is false, flushes all entries.
*/
static bool
-pgstat_flush_pending_entries(bool nowait)
+pgstat_flush_pending_entries(bool nowait, bool anytime_only)
{
bool have_pending = false;
dlist_node *cur = NULL;
@@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait)
Assert(!kind_info->fixed_amount);
Assert(kind_info->flush_pending_cb != NULL);
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ {
+ have_pending = true;
+
+ if (dlist_has_next(&pgStatPending, cur))
+ next = dlist_next_node(&pgStatPending, cur);
+ else
+ next = NULL;
+
+ cur = next;
+ continue;
+ }
+
/* flush the stats, if possible */
- did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
+ did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
@@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait)
return have_pending;
}
+/*
+ * Flush fixed-amount stats.
+ *
+ * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions).
+ * If anytime_only is false, flushes all stats with flush_static_cb.
+ */
+static bool
+pgstat_flush_fixed_stats(bool nowait, bool anytime_only)
+{
+ bool partial_flush = false;
+
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->flush_static_cb)
+ continue;
+
+ /* Skip transactional stats if we're in anytime_only mode */
+ if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY)
+ continue;
+
+ partial_flush |= kind_info->flush_static_cb(nowait, anytime_only);
+ }
+
+ return partial_flush;
+}
/* ------------------------------------------------------------
* Helper / infrastructure functions
@@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra)
if (pgstat_fetch_consistency != newval)
force_stats_snapshot_clear = true;
}
+
+/*
+ * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
+ * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
+ * Safe to call inside transactions.
+ */
+void
+pgstat_report_anytime_stat(bool force)
+{
+ bool nowait = !force;
+
+ pgstat_assert_is_up();
+
+ /* Flush stats outside of transaction boundary */
+ pgstat_flush_pending_entries(nowait, true);
+ pgstat_flush_fixed_stats(nowait, true);
+}
+
+/*
+ * Timeout handler for flushing non-transactional stats.
+ */
+void
+AnytimeStatsUpdateTimeoutHandler(void)
+{
+ AnytimeStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 1350f5f62f1..b0bd220882d 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -31,6 +31,7 @@
#include "storage/procarray.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
/*
* Backend statistics counts waiting to be flushed out. These counters may be
@@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context,
INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op],
io_time);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context,
PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt;
PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes;
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
backend_has_iostats = true;
pgstat_report_fixed = true;
}
@@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)
* if some statistics could not be flushed due to lock contention.
*/
bool
-pgstat_flush_backend(bool nowait, bits32 flags)
+pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only)
{
PgStat_EntryRef *entry_ref;
bool has_pending_data = false;
@@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
* If some stats could not be flushed due to lock contention, return true.
*/
bool
-pgstat_backend_flush_cb(bool nowait)
+pgstat_backend_flush_cb(bool nowait, bool anytime_only)
{
- return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL);
+ return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only);
}
/*
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index ed2fd801189..1c5f0c3ec40 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -61,7 +61,7 @@ pgstat_report_bgwriter(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 1f70194b7a7..2d89a082464 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -68,7 +68,7 @@ pgstat_report_checkpointer(void)
/*
* Report IO statistics
*/
- pgstat_flush_io(false);
+ pgstat_flush_io(false, true);
}
/*
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index d7f6d4c5ee6..61094f96a6c 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -425,7 +425,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStatShared_Database *sharedent;
PgStat_StatDBEntry *pendingent;
diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c
index e6b84283c6c..5ba4958382f 100644
--- a/src/backend/utils/activity/pgstat_function.c
+++ b/src/backend/utils/activity/pgstat_function.c
@@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_FunctionCounts *localent;
PgStatShared_Function *shfuncent;
+ Assert(!anytime_only);
+
localent = (PgStat_FunctionCounts *) entry_ref->pending;
shfuncent = (PgStatShared_Function *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 28de24538dc..7cd32900236 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -19,6 +19,7 @@
#include "executor/instrument.h"
#include "storage/bufmgr.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
static PgStat_PendingIO PendingIOStats;
static bool have_iostats = false;
@@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op,
/* Add the per-backend counts */
pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes);
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_iostats = true;
pgstat_report_fixed = true;
}
@@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void)
* Simpler wrapper of pgstat_io_flush_cb()
*/
void
-pgstat_flush_io(bool nowait)
+pgstat_flush_io(bool nowait, bool anytime_only)
{
- (void) pgstat_io_flush_cb(nowait);
+ (void) pgstat_io_flush_cb(nowait, anytime_only);
}
/*
@@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait)
* acquired. Otherwise, return false.
*/
bool
-pgstat_io_flush_cb(bool nowait)
+pgstat_io_flush_cb(bool nowait, bool anytime_only)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..04d21483d93 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
* is done -- which will likely vacuum many relations -- or until the
* VACUUM command has processed all tables and committed.
*/
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel,
pgstat_unlock_entry(entry_ref);
/* see pgstat_report_vacuum() */
- pgstat_flush_io(false);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(false, true);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
* entry when successfully flushing.
*/
bool
-pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
Oid dboid;
PgStat_TableStatus *lstats; /* pending stats entry */
@@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
+ Assert(!anytime_only);
+
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 2190f388eae..bf8a4d58673 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -19,6 +19,7 @@
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
+#include "utils/timeout.h"
static inline PgStat_SLRUStats *get_slru_entry(int slru_idx);
@@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name)
* acquired. Otherwise return false.
*/
bool
-pgstat_slru_flush_cb(bool nowait)
+pgstat_slru_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
int i;
@@ -223,6 +224,9 @@ get_slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
+ /* Schedule next anytime stats update timeout */
+ pgstat_schedule_anytime_update();
+
have_slrustats = true;
pgstat_report_fixed = true;
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index 500b1899188..c4614817966 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid)
* false without flushing the entry. Otherwise returns true.
*/
bool
-pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_BackendSubEntry *localent;
PgStatShared_Subscription *shsubent;
+ Assert(!anytime_only);
+
localent = (PgStat_BackendSubEntry *) entry_ref->pending;
shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats;
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 183e0a7a97b..eb5f8f46925 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -51,12 +51,12 @@ pgstat_report_wal(bool force)
nowait = !force;
/* flush wal stats */
- (void) pgstat_wal_flush_cb(nowait);
- pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);
+ (void) pgstat_wal_flush_cb(nowait, true);
+ pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
/* flush IO stats */
- pgstat_flush_io(nowait);
- (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO);
+ pgstat_flush_io(nowait, true);
+ (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -88,7 +88,7 @@ pgstat_wal_have_pending(void)
* acquired. Otherwise return false.
*/
bool
-pgstat_wal_flush_cb(bool nowait)
+pgstat_wal_flush_cb(bool nowait, bool anytime_only)
{
PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
WalUsage wal_usage_diff = {0};
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..ad44826c39e 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
+volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..f45365f47f7 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -64,6 +64,7 @@
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
+#include "utils/pgstat_internal.h"
#include "utils/portal.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -765,6 +766,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler);
RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
IdleStatsUpdateTimeoutHandler);
+ RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT,
+ AnytimeStatsUpdateTimeoutHandler);
}
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..8aeb9628871 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fff7ecc2533..b340a680614 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,9 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
+/* Minimum interval non-forced stats flushes */
+#define PGSTAT_MIN_INTERVAL 1000
+
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -533,8 +536,21 @@ extern void pgstat_initialize(void);
/* Functions called from backends */
extern long pgstat_report_stat(bool force);
+extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
+/*
+ * Schedule the next anytime stats update timeout.
+ *
+ * This should be called whenever accumulating statistics that support
+ * FLUSH_ANYTIME flushing mode.
+ */
+#define pgstat_schedule_anytime_update() \
+ do { \
+ if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ } while (0)
+
extern void pgstat_reset_counters(void);
extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_reset_of_kind(PgStat_Kind kind);
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 9b8fbae00ed..607f4255268 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus
PgStat_TableXactStatus *first; /* head of list for this subxact */
} PgStat_SubXactStatus;
+/*
+ * Flush mode for statistics kinds.
+ *
+ * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the
+ * default value.
+ */
+typedef enum PgStat_FlushMode
+{
+ FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at
+ * transaction boundary */
+ FLUSH_ANYTIME, /* All fields can be flushed anytime,
+ * including within transactions */
+} PgStat_FlushMode;
/*
* Metadata for a specific kind of statistics.
@@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo
*/
bool track_entry_count:1;
+ /*
+ * The mode of when to flush stats. See PgStat_FlushMode for more details.
+ *
+ * This member only has meaning for statistics kinds that accumulate
+ * pending stats and use flush callbacks. For kinds that write directly to
+ * shared memory (e.g., archiver, bgwriter, checkpointer), this member has
+ * no effect.
+ */
+ PgStat_FlushMode flush_mode;
+
/*
* The size of an entry in the shared stats hash table (pointed to by
* PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
@@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo
* For variable-numbered stats: flush pending stats. Required if pending
* data is used. See flush_static_cb when dealing with stats data that
* that cannot use PgStat_EntryRef->pending.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait);
+ bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only);
/*
* For variable-numbered stats: delete pending stats. Optional.
@@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo
*
* "pgstat_report_fixed" needs to be set to trigger the flush of pending
* stats.
+ *
+ * The anytime_only parameter indicates whether this is an anytime flush.
*/
- bool (*flush_static_cb) (bool nowait);
+ bool (*flush_static_cb) (bool nowait, bool anytime_only);
/*
* For fixed-numbered statistics: Reset All.
@@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind,
extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid);
extern void pgstat_snapshot_fixed(PgStat_Kind kind);
+extern void AnytimeStatsUpdateTimeoutHandler(void);
/*
@@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void);
#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */
#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
-extern bool pgstat_flush_backend(bool nowait, bits32 flags);
-extern bool pgstat_backend_flush_cb(bool nowait);
+extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only);
+extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header,
TimestampTz ts);
@@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel);
extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid);
extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts);
-extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_function.c
*/
-extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time
* Functions in pgstat_io.c
*/
-extern void pgstat_flush_io(bool nowait);
+extern void pgstat_flush_io(bool nowait, bool anytime_only);
-extern bool pgstat_io_flush_cb(bool nowait);
+extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool
extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state);
-extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref);
extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
@@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
* Functions in pgstat_slru.c
*/
-extern bool pgstat_slru_flush_cb(bool nowait);
+extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void);
*/
extern void pgstat_wal_init_backend_cb(void);
-extern bool pgstat_wal_flush_cb(bool nowait);
+extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only);
extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
@@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void);
* Functions in pgstat_subscription.c
*/
-extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only);
extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts);
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 0965b590b34..10723bb664c 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -35,6 +35,7 @@ typedef enum TimeoutId
IDLE_SESSION_TIMEOUT,
IDLE_STATS_UPDATE_TIMEOUT,
CLIENT_CONNECTION_CHECK_TIMEOUT,
+ ANYTIME_STATS_UPDATE_TIMEOUT,
STARTUP_PROGRESS_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index 64a8fe63cce..bc0b5d6e0eb 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL;
/* Flush callback: merge pending stats into shared memory */
static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref,
- bool nowait);
+ bool nowait, bool anytime_only);
/* Serialization callback: write auxiliary entry data */
static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key,
@@ -150,7 +150,7 @@ _PG_init(void)
* Returns false only if nowait=true and lock acquisition fails.
*/
static bool
-test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait)
+test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
{
PgStat_StatCustomVarEntry *pending_entry;
PgStatShared_CustomVarEntry *shared_entry;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9f5ee8fd482..860f835c088 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_Counter
PgStat_EntryRef
PgStat_EntryRefHashEntry
PgStat_FetchConsistency
+PgStat_FlushMode
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_HashKey
--
2.34.1
[text/x-diff] v6-0002-Add-anytime-flush-tests-for-custom-stats.patch (9.3K, 3-v6-0002-Add-anytime-flush-tests-for-custom-stats.patch)
download | inline diff:
From c381419eb6d5018c3fe5476ac6edae8673ac5343 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Thu, 5 Feb 2026 05:54:34 +0000
Subject: [PATCH v6 2/5] Add anytime flush tests for custom stats
---
.../test_custom_stats/t/001_custom_stats.pl | 41 ++++++++++++
.../test_custom_fixed_stats--1.0.sql | 10 +++
.../test_custom_fixed_stats.c | 66 +++++++++++++++++++
.../test_custom_var_stats--1.0.sql | 5 ++
.../test_custom_stats/test_custom_var_stats.c | 27 ++++++++
5 files changed, 149 insertions(+)
31.4% src/test/modules/test_custom_stats/t/
68.5% src/test/modules/test_custom_stats/
diff --git a/src/test/modules/test_custom_stats/t/001_custom_stats.pl b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
index 9e6a7a38577..36d9fc3fde1 100644
--- a/src/test/modules/test_custom_stats/t/001_custom_stats.pl
+++ b/src/test/modules/test_custom_stats/t/001_custom_stats.pl
@@ -156,5 +156,46 @@ $result = $node->safe_psql('postgres',
);
is($result, "0", "report of fixed-sized after manual reset");
+# Test FLUSH_ANYTIME mechanism with custom fixed stats
+# This verifies that custom stats can be flushed during a transaction
+
+# Reset stats first
+$node->safe_psql('postgres', q(select test_custom_stats_fixed_reset()));
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+my $anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
+ -- Force anytime flush (inside transaction!)
+ select pg_stat_force_anytime_flush();
+ -- Check
+ select 'anytime:'||numcalls from test_custom_stats_fixed_report();
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^anytime:2/m,
+ "anytime fixed stats flushed during transaction");
+
+# Test FLUSH_ANYTIME mechanism with custom variable stats
+# This verifies that custom stats can be flushed during a transaction
+
+$node->safe_psql('postgres', q(select pg_stat_force_next_flush()));
+
+$anytime_test = q[
+ BEGIN;
+ -- Accumulate stats
+ select test_custom_stats_var_anytime_update('entry2');
+ select test_custom_stats_var_anytime_update('entry2');
+ -- Force anytime flush (inside transaction!)
+ select pg_stat_force_anytime_flush();
+ -- Check
+ select * from test_custom_stats_var_report('entry2');
+];
+
+$result = $node->safe_psql('postgres', $anytime_test);
+like($result, qr/^entry2|2|/m,
+ "anytime var stats flushed during transaction");
+
# Test completed successfully
done_testing();
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
index 69a93b5241f..c0a418c3ae3 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats--1.0.sql
@@ -18,3 +18,13 @@ CREATE FUNCTION test_custom_stats_fixed_reset()
RETURNS void
AS 'MODULE_PATHNAME', 'test_custom_stats_fixed_reset'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_fixed_anytime_update()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION pg_stat_force_anytime_flush()
+RETURNS void
+AS 'MODULE_PATHNAME'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
index 908bd18a7c7..6b3bc3257ab 100644
--- a/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_fixed_stats.c
@@ -18,6 +18,7 @@
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_fixed_stats",
@@ -43,11 +44,13 @@ typedef struct PgStatShared_CustomFixedEntry
static void test_custom_stats_fixed_init_shmem_cb(void *stats);
static void test_custom_stats_fixed_reset_all_cb(TimestampTz ts);
static void test_custom_stats_fixed_snapshot_cb(void);
+static bool test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only);
static const PgStat_KindInfo custom_stats = {
.name = "test_custom_fixed_stats",
.fixed_amount = true, /* exactly one entry */
.write_to_file = true, /* persist to stats file */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.shared_size = sizeof(PgStat_StatCustomFixedEntry),
.shared_data_off = offsetof(PgStatShared_CustomFixedEntry, stats),
@@ -56,8 +59,12 @@ static const PgStat_KindInfo custom_stats = {
.init_shmem_cb = test_custom_stats_fixed_init_shmem_cb,
.reset_all_cb = test_custom_stats_fixed_reset_all_cb,
.snapshot_cb = test_custom_stats_fixed_snapshot_cb,
+ .flush_static_cb = test_custom_stats_fixed_flush_cb,
};
+/* Pending statistics */
+static PgStat_StatCustomFixedEntry PendingCustomStats = {0};
+
/*
* Kind ID for test_custom_fixed_stats.
*/
@@ -141,6 +148,38 @@ test_custom_stats_fixed_snapshot_cb(void)
#undef FIXED_COMP
}
+/*
+ * test_custom_stats_fixed_flush_cb
+ * Flush pending stats to shared memory
+ */
+static bool
+test_custom_stats_fixed_flush_cb(bool nowait, bool anytime_only)
+{
+ PgStatShared_CustomFixedEntry *stats_shmem;
+
+ /* Nothing to flush if no calls were made */
+ if (PendingCustomStats.numcalls == 0)
+ return false;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_TEST_CUSTOM_FIXED_STATS);
+
+ if (nowait && !LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true; /* failed to flush */
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numcalls += PendingCustomStats.numcalls;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+
+ LWLockRelease(&stats_shmem->lock);
+
+ /* Reset pending stats */
+ PendingCustomStats.numcalls = 0;
+
+ return false; /* successfully flushed */
+}
+
/*--------------------------------------------------------------------------
* SQL-callable functions
*--------------------------------------------------------------------------
@@ -222,3 +261,30 @@ test_custom_stats_fixed_report(PG_FUNCTION_ARGS)
/* Return as tuple */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/*
+ * test_custom_stats_fixed_anytime_update
+ * Increment call counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_fixed_anytime_update);
+Datum
+test_custom_stats_fixed_anytime_update(PG_FUNCTION_ARGS)
+{
+ /* Accumulate in pending stats */
+ PendingCustomStats.numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+ pgstat_report_fixed = true;
+
+ PG_RETURN_VOID();
+}
+
+/* Helper function for testing ANYTIME flush */
+PG_FUNCTION_INFO_V1(pg_stat_force_anytime_flush);
+Datum
+pg_stat_force_anytime_flush(PG_FUNCTION_ARGS)
+{
+ pgstat_report_anytime_stat(true);
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
index 5ed8cfc2dcf..ed66d38981e 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats--1.0.sql
@@ -24,3 +24,8 @@ CREATE FUNCTION test_custom_stats_var_report(INOUT name TEXT,
RETURNS SETOF record
AS 'MODULE_PATHNAME', 'test_custom_stats_var_report'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+CREATE FUNCTION test_custom_stats_var_anytime_update(IN name TEXT)
+RETURNS void
+AS 'MODULE_PATHNAME', 'test_custom_stats_var_anytime_update'
+LANGUAGE C STRICT PARALLEL UNSAFE;
diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c
index bc0b5d6e0eb..207e841911b 100644
--- a/src/test/modules/test_custom_stats/test_custom_var_stats.c
+++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c
@@ -17,6 +17,7 @@
#include "storage/dsm_registry.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+#include "utils/timeout.h"
PG_MODULE_MAGIC_EXT(
.name = "test_custom_var_stats",
@@ -107,6 +108,7 @@ static const PgStat_KindInfo custom_stats = {
.name = "test_custom_var_stats",
.fixed_amount = false, /* variable number of entries */
.write_to_file = true, /* persist across restarts */
+ .flush_mode = FLUSH_ANYTIME, /* can be flushed anytime */
.track_entry_count = true, /* count active entries */
.accessed_across_databases = true, /* global statistics */
.shared_size = sizeof(PgStatShared_CustomVarEntry),
@@ -689,3 +691,28 @@ test_custom_stats_var_report(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * test_custom_stats_var_anytime_update
+ * Increment custom statistic counter and schedule anytime flush
+ */
+PG_FUNCTION_INFO_V1(test_custom_stats_var_anytime_update);
+Datum
+test_custom_stats_var_anytime_update(PG_FUNCTION_ARGS)
+{
+ char *stat_name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_EntryRef *entry_ref;
+ PgStat_StatCustomVarEntry *pending_entry;
+
+ /* Get pending entry in local memory */
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, InvalidOid,
+ PGSTAT_CUSTOM_VAR_STATS_IDX(stat_name), NULL);
+
+ pending_entry = (PgStat_StatCustomVarEntry *) entry_ref->pending;
+ pending_entry->numcalls++;
+
+ /* Schedule anytime stats update */
+ pgstat_schedule_anytime_update();
+
+ PG_RETURN_VOID();
+}
--
2.34.1
[text/x-diff] v6-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch (7.6K, 4-v6-0003-Add-GUC-to-specify-non-transactional-statistics-f.patch)
download | inline diff:
From 69f3e4f063a956f429ee0dbdbd213dcad8242613 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Wed, 28 Jan 2026 07:53:13 +0000
Subject: [PATCH v6 3/5] Add GUC to specify non-transactional statistics flush
interval
Adding pgstat_flush_interval, a new GUC to set the interval between flushes of
non-transactional statistics.
---
doc/src/sgml/config.sgml | 34 +++++++++++++++++++
src/backend/utils/activity/pgstat.c | 16 +++++++++
src/backend/utils/misc/guc_parameters.dat | 10 ++++++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/pgstat.h | 6 ++--
src/include/utils/guc_hooks.h | 1 +
6 files changed, 64 insertions(+), 4 deletions(-)
59.1% doc/src/sgml/
14.2% src/backend/utils/activity/
14.5% src/backend/utils/misc/
12.0% src/include/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f1af1505cf3..20666679f90 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8871,6 +8871,40 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-stats-flush-interval" xreflabel="stats_flush_interval">
+ <term><varname>stats_flush_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>stats_flush_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Sets the interval at which statistics that can be updated while a
+ transaction is still running are made visible. These include, for example,
+ WAL activity and I/O operations.
+ Such statistics are refreshed at the specified interval and can be observed
+ during active transactions in monitoring views such as
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>
+ and
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>.
+ Other statistics are only made visible at transaction end and are not
+ affected by this setting.
+ If the value is specified without a unit, milliseconds are assumed.
+ The default is 10 seconds (<literal>10s</literal>), which is generally
+ the smallest practical value for long-running transactions.
+ </para>
+ <note>
+ <para>
+ This parameter does not affect statistics that are only reported at
+ transaction end, such as the columns of <structname>pg_stat_all_tables</structname>
+ (for example, <structfield>n_tup_ins</structfield>, <structfield>n_tup_upd</structfield>,
+ and <structfield>n_tup_del</structfield>). These statistics are always
+ flushed at the end of a transaction.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 411b65aae3e..79eb59b5625 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -123,6 +123,8 @@
* ----------
*/
+/* minimum interval non-forced stats flushes.*/
+#define PGSTAT_MIN_INTERVAL 1000
/* how long until to block flushing pending stats updates */
#define PGSTAT_MAX_INTERVAL 60000
/* when to call pgstat_report_stat() again, even when idle */
@@ -203,6 +205,7 @@ static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
bool pgstat_track_counts = false;
int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_CACHE;
+int pgstat_flush_interval = 10000;
/* ----------
@@ -2164,6 +2167,19 @@ assign_stats_fetch_consistency(int newval, void *extra)
force_stats_snapshot_clear = true;
}
+/*
+ * GUC assign_hook for stats_flush_interval.
+ */
+void
+assign_stats_flush_interval(int newval, void *extra)
+{
+ if (get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT))
+ {
+ disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false);
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, newval);
+ }
+}
+
/*
* Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional
* stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary.
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index c1f1603cd39..fc0e4259b36 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -2789,6 +2789,16 @@
assign_hook => 'assign_stats_fetch_consistency',
},
+{ name => 'stats_flush_interval', type => 'int', context => 'PGC_USERSET', group => 'STATS_CUMULATIVE',
+ short_desc => 'Sets the interval between flushes of non-transactional statistics.',
+ flags => 'GUC_UNIT_MS',
+ variable => 'pgstat_flush_interval',
+ boot_val => '10000',
+ min => '1000',
+ max => 'INT_MAX',
+ assign_hook => 'assign_stats_flush_interval'
+},
+
{ name => 'subtransaction_buffers', type => 'int', context => 'PGC_POSTMASTER', group => 'RESOURCES_MEM',
short_desc => 'Sets the size of the dedicated buffer pool used for the subtransaction cache.',
long_desc => '0 means use a fraction of "shared_buffers".',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1ae594af843..3f998a0bea0 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -673,6 +673,7 @@
#track_wal_io_timing = off
#track_functions = none # none, pl, all
#stats_fetch_consistency = cache # cache, none, snapshot
+#stats_flush_interval = 10s # in milliseconds
# - Monitoring -
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b340a680614..ef856dbf55b 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,9 +35,6 @@
/* Default directory to store temporary statistics data in */
#define PG_STAT_TMP_DIR "pg_stat_tmp"
-/* Minimum interval non-forced stats flushes */
-#define PGSTAT_MIN_INTERVAL 1000
-
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
{
@@ -548,7 +545,7 @@ extern void pgstat_force_next_flush(void);
#define pgstat_schedule_anytime_update() \
do { \
if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \
- enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \
+ enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, pgstat_flush_interval); \
} while (0)
extern void pgstat_reset_counters(void);
@@ -828,6 +825,7 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_fetch_consistency;
+extern PGDLLIMPORT int pgstat_flush_interval;
/*
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index b6ecb0e769f..3a2ae6c41cd 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -132,6 +132,7 @@ extern bool check_session_authorization(char **newval, void **extra, GucSource s
extern void assign_session_authorization(const char *newval, void *extra);
extern void assign_session_replication_role(int newval, void *extra);
extern void assign_stats_fetch_consistency(int newval, void *extra);
+extern void assign_stats_flush_interval(int newval, void *extra);
extern bool check_ssl(bool *newval, void **extra, GucSource source);
extern bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_standard_conforming_strings(bool *newval, void **extra,
--
2.34.1
[text/x-diff] v6-0004-Remove-useless-calls-to-flush-some-stats.patch (7.6K, 5-v6-0004-Remove-useless-calls-to-flush-some-stats.patch)
download | inline diff:
From de9afb385c171f3e0f8de2a0dbe4f72143216686 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Tue, 6 Jan 2026 11:06:31 +0000
Subject: [PATCH v6 4/5] Remove useless calls to flush some stats
Now that some stats can be flushed outside of transaction boundaries, remove
useless calls to report/flush some stats. Those calls were in place because
before commit <XXXX> stats were flushed only at transaction boundaries.
Note that:
- it reverts 039549d70f6 (it just keeps its tests)
- it can't be done for checkpointer and bgworker for example because they don't
have a flush callback to call
- it can't be done for auxiliary process (walsummarizer for example) because they
currently do not register the new timeout handler
---
src/backend/replication/walreceiver.c | 10 ------
src/backend/replication/walsender.c | 36 ++------------------
src/backend/utils/activity/pgstat_relation.c | 13 -------
src/test/recovery/t/001_stream_rep.pl | 1 +
src/test/subscription/t/001_rep_changes.pl | 1 +
5 files changed, 4 insertions(+), 57 deletions(-)
69.4% src/backend/replication/
23.4% src/backend/utils/activity/
3.4% src/test/recovery/t/
3.5% src/test/subscription/t/
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b7c114d3b..953ba97ed00 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -571,16 +571,6 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
*/
bool requestReply = false;
- /*
- * Report pending statistics to the cumulative stats
- * system. This location is useful for the report as it
- * is not within a tight loop in the WAL receiver, to
- * avoid bloating pgstats with requests, while also making
- * sure that the reports happen each time a status update
- * is sent.
- */
- pgstat_report_wal(false);
-
/*
* Check if time since last receive from primary has
* reached the configured limit.
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a7214d0dc6f..9a136e35b48 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -94,14 +94,10 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
-#include "utils/pgstat_internal.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
-/* Minimum interval used by walsender for stats flushes, in ms */
-#define WALSENDER_STATS_FLUSH_INTERVAL 1000
-
/*
* Maximum data payload in a WAL data message. Must be >= XLOG_BLCKSZ.
*
@@ -1846,7 +1842,6 @@ WalSndWaitForWal(XLogRecPtr loc)
int wakeEvents;
uint32 wait_event = 0;
static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr;
- TimestampTz last_flush = 0;
/*
* Fast path to avoid acquiring the spinlock in case we already know we
@@ -1867,7 +1862,6 @@ WalSndWaitForWal(XLogRecPtr loc)
{
bool wait_for_standby_at_stop = false;
long sleeptime;
- TimestampTz now;
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
@@ -1973,8 +1967,7 @@ WalSndWaitForWal(XLogRecPtr loc)
* new WAL to be generated. (But if we have nothing to send, we don't
* want to wake on socket-writable.)
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
wakeEvents = WL_SOCKET_READABLE;
@@ -1983,15 +1976,6 @@ WalSndWaitForWal(XLogRecPtr loc)
Assert(wait_event != 0);
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
WalSndWait(wakeEvents, sleeptime, wait_event);
}
@@ -2894,8 +2878,6 @@ WalSndCheckTimeOut(void)
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
- TimestampTz last_flush = 0;
-
/*
* Initialize the last reply timestamp. That enables timeout processing
* from hereon.
@@ -2985,9 +2967,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
* WalSndWaitForWal() handle any other blocking; idle receivers need
* its additional actions. For physical replication, also block if
* caught up; its send_data does not block.
- *
- * The IO statistics are reported in WalSndWaitForWal() for the
- * logical WAL senders.
*/
if ((WalSndCaughtUp && send_data != XLogSendLogical &&
!streamingDoneSending) ||
@@ -2995,7 +2974,6 @@ WalSndLoop(WalSndSendDataCallback send_data)
{
long sleeptime;
int wakeEvents;
- TimestampTz now;
if (!streamingDoneReceiving)
wakeEvents = WL_SOCKET_READABLE;
@@ -3006,21 +2984,11 @@ WalSndLoop(WalSndSendDataCallback send_data)
* Use fresh timestamp, not last_processing, to reduce the chance
* of reaching wal_sender_timeout before sending a keepalive.
*/
- now = GetCurrentTimestamp();
- sleeptime = WalSndComputeSleeptime(now);
+ sleeptime = WalSndComputeSleeptime(GetCurrentTimestamp());
if (pq_is_send_pending())
wakeEvents |= WL_SOCKET_WRITEABLE;
- /* Report IO statistics, if needed */
- if (TimestampDifferenceExceeds(last_flush, now,
- WALSENDER_STATS_FLUSH_INTERVAL))
- {
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
- last_flush = now;
- }
-
/* Sleep until something happens or we time out */
WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
}
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 04d21483d93..ae2952cae89 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -260,15 +260,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
}
pgstat_unlock_entry(entry_ref);
-
- /*
- * Flush IO statistics now. pgstat_report_stat() will flush IO stats,
- * however this will not be called until after an entire autovacuum cycle
- * is done -- which will likely vacuum many relations -- or until the
- * VACUUM command has processed all tables and committed.
- */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
@@ -360,10 +351,6 @@ pgstat_report_analyze(Relation rel,
}
pgstat_unlock_entry(entry_ref);
-
- /* see pgstat_report_vacuum() */
- pgstat_flush_io(false, true);
- (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true);
}
/*
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index e9ac67813c7..c058a5f9b1f 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,7 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role' => 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 7d41715ed81..bceec2adede 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -11,6 +11,7 @@ use Test::More;
# Initialize publisher node
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
$node_publisher->start;
# Create subscriber node
--
2.34.1
[text/x-diff] v6-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch (26.2K, 6-v6-0005-Change-RELATION-and-DATABASE-stats-to-anytime-flu.patch)
download | inline diff:
From 5a57a7034acf37b11f9020e415c801963270bd33 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Mon, 19 Jan 2026 06:27:55 +0000
Subject: [PATCH v6 5/5] Change RELATION and DATABASE stats to anytime flush
This commit allows mixing fields with different transaction behavior within
the same RELATION or DATABASE statistics kind: some fields are transactional
(e.g., tuple inserts/updates/deletes) while others are non-transactional
(e.g., sequential scans, blocks read).
It modifies the relation flush callback to handle the anytime_only parameter
introduced in commit <nnnn>.
Implementation details:
- Change RELATION from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Change DATABASE from FLUSH_AT_TXN_BOUNDARY to FLUSH_ANYTIME
- Modify pgstat_relation_flush_cb() to handle anytime_only parameter: when
true, then flush only non-transactional stats and when false, then flush all
the stats. When set to true, it clears flushed fields from pending stats to
prevent double-counting at transaction boundary
DATABASE stats inherit the anytime flush behavior so that relation-derived
stats (tuples_returned, tuples_fetched, blocks_fetched, blocks_hit) are
visible while transactions are in progress.
Tests are added to verify the anytime flush behavior for mixed fields.
---
doc/src/sgml/monitoring.sgml | 37 ++++++-
src/backend/utils/activity/pgstat.c | 17 ++--
src/backend/utils/activity/pgstat_relation.c | 86 ++++++++++++----
src/include/pgstat.h | 27 ++++-
src/test/isolation/expected/stats.out | 102 +++++++++++++++++++
src/test/isolation/expected/stats_1.out | 102 +++++++++++++++++++
src/test/isolation/specs/stats.spec | 27 ++++-
7 files changed, 368 insertions(+), 30 deletions(-)
13.0% doc/src/sgml/
25.9% src/backend/utils/activity/
6.0% src/include/
49.8% src/test/isolation/expected/
5.1% src/test/isolation/specs/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b77d189a500..aa7bd2e2e2a 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3767,6 +3767,19 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ Some statistics are updated while a transaction is in progress (for example,
+ <structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
+ <structfield>tup_returned</structfield> and <structfield>tup_fetched</structfield>).
+ Statistics that either do not depend on transactions or require transactional
+ consistency are updated only when the transaction ends. Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>, <structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and <structfield>tup_deleted</structfield>.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-database-conflicts-view">
@@ -3956,8 +3969,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_seq_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last sequential scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last sequential scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -3984,8 +3997,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>last_idx_scan</structfield> <type>timestamp with time zone</type>
</para>
<para>
- The time of the last index scan on this table, based on the
- most recent transaction stop time
+ The approximate time of the last index scan on this table, updated
+ at least every <varname>stats_flush_interval</varname>
</para></entry>
</row>
@@ -4223,6 +4236,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</tgroup>
</table>
+ <note>
+ <para>
+ The <structfield>seq_scan</structfield>, <structfield>last_seq_scan</structfield>,
+ <structfield>seq_tup_read</structfield>, <structfield>idx_scan</structfield>,
+ <structfield>last_idx_scan</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
+
</sect2>
<sect2 id="monitoring-pg-stat-all-indexes-view">
@@ -4404,6 +4426,13 @@ description | Waiting for a newly initialized WAL file to reach durable storage
tuples (see <xref linkend="indexes-multicolumn"/>).
</para>
</note>
+ <note>
+ <para>
+ The <structfield>idx_scan</structfield>, <structfield>last_idx_scan</structfield>,
+ <structfield>idx_tup_read</structfield> and <structfield>idx_tup_fetch</structfield>
+ are updated while the transactions are in progress.
+ </para>
+ </note>
<tip>
<para>
<command>EXPLAIN ANALYZE</command> outputs the total number of index
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 79eb59b5625..9234185de64 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -291,7 +291,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
/* so pg_stat_database entries can be seen in all databases */
.accessed_across_databases = true,
@@ -309,7 +309,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE]
.fixed_amount = false,
.write_to_file = true,
- .flush_mode = FLUSH_AT_TXN_BOUNDARY,
+ .flush_mode = FLUSH_ANYTIME,
.shared_size = sizeof(PgStatShared_Relation),
.shared_data_off = offsetof(PgStatShared_Relation, stats),
@@ -1344,10 +1344,12 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref)
dlist_delete(&entry_ref->pending_node);
}
+
/*
* Flush out pending variable-numbered stats.
*
- * If anytime_only is true, only flushes FLUSH_ANYTIME entries.
+ * If anytime_only is true, only flushes FLUSH_ANYTIME entries. For entries
+ * that support it, the callback may flush only non-transactional fields.
* This is safe to call inside transactions.
*
* If anytime_only is false, flushes all entries.
@@ -1378,6 +1380,7 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
PgStat_Kind kind = key.kind;
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
bool did_flush;
+ bool is_partial_flush = false;
dlist_node *next;
Assert(!kind_info->fixed_amount);
@@ -1397,19 +1400,21 @@ pgstat_flush_pending_entries(bool nowait, bool anytime_only)
continue;
}
- /* flush the stats, if possible */
did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only);
Assert(did_flush || nowait);
+ /* Partial flush only happens in anytime mode for FLUSH_ANYTIME stats */
+ is_partial_flush = (anytime_only && kind_info->flush_mode == FLUSH_ANYTIME);
+
/* determine next entry, before deleting the pending entry */
if (dlist_has_next(&pgStatPending, cur))
next = dlist_next_node(&pgStatPending, cur);
else
next = NULL;
- /* if successfully flushed, remove entry */
- if (did_flush)
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
else
have_pending = true;
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ae2952cae89..10ab9e4cc76 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -47,7 +47,19 @@ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_lev
static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info);
static void save_truncdrop_counters(PgStat_TableXactStatus *trans, bool is_drop);
static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
+static void flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry,
+ PgStat_TableCounts *counts, bool anytime_only);
+/*
+ * Update database statistics with non-transactional stats.
+ */
+#define UPDATE_DATABASE_ANYTIME_STATS(dbentry, counts) \
+ do { \
+ (dbentry)->tuples_returned += (counts)->tuples_returned; \
+ (dbentry)->tuples_fetched += (counts)->tuples_fetched; \
+ (dbentry)->blocks_fetched += (counts)->blocks_fetched; \
+ (dbentry)->blocks_hit += (counts)->blocks_hit; \
+ } while (0)
/*
* Copy stats between relations. This is used for things like REINDEX
@@ -789,6 +801,29 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Helper function to flush non-transactional statistics.
+ */
+static void
+flush_relation_anytime_stats(PgStat_StatTabEntry *tabentry, PgStat_TableCounts *counts,
+ bool anytime_only)
+{
+ TimestampTz t;
+
+ tabentry->numscans += counts->numscans;
+ if (counts->numscans)
+ {
+ t = anytime_only ? GetCurrentTimestamp() : GetCurrentTransactionStopTimestamp();
+ if (t > tabentry->lastscan)
+ tabentry->lastscan = t;
+ }
+
+ tabentry->tuples_returned += counts->tuples_returned;
+ tabentry->tuples_fetched += counts->tuples_fetched;
+ tabentry->blocks_fetched += counts->blocks_fetched;
+ tabentry->blocks_hit += counts->blocks_hit;
+}
+
/*
* Flush out pending stats for the entry
*
@@ -797,6 +832,13 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
*
* Some of the stats are copied to the corresponding pending database stats
* entry when successfully flushing.
+ *
+ * If anytime_only is true, only non-transactional fields are flushed
+ * (numscans, tuples_returned, tuples_fetched, blocks_fetched, blocks_hit).
+ * Transactional fields remain pending until transaction boundary.
+ *
+ * Some of the stats are copied to the corresponding pending database stats
+ * entry when successfully flushing.
*/
bool
pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only)
@@ -807,8 +849,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
PgStat_StatTabEntry *tabentry; /* table entry of shared stats */
PgStat_StatDBEntry *dbentry; /* pending database entry */
- Assert(!anytime_only);
-
dboid = entry_ref->shared_entry->key.dboid;
lstats = (PgStat_TableStatus *) entry_ref->pending;
shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats;
@@ -824,19 +864,36 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
if (!pgstat_lock_entry(entry_ref, nowait))
return false;
- /* add the values to the shared entry. */
tabentry = &shtabstats->stats;
- tabentry->numscans += lstats->counts.numscans;
- if (lstats->counts.numscans)
+ if (anytime_only)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
- if (t > tabentry->lastscan)
- tabentry->lastscan = t;
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, true);
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* Also update the corresponding fields in database stats */
+ dbentry = pgstat_prep_database_pending(dboid);
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
+ /*
+ * Clear the flushed fields from pending stats to prevent
+ * double-counting when we flush all fields at transaction boundary.
+ */
+ lstats->counts.numscans = 0;
+ lstats->counts.tuples_returned = 0;
+ lstats->counts.tuples_fetched = 0;
+ lstats->counts.blocks_fetched = 0;
+ lstats->counts.blocks_hit = 0;
+
+ return true;
}
- tabentry->tuples_returned += lstats->counts.tuples_returned;
- tabentry->tuples_fetched += lstats->counts.tuples_fetched;
+
+ /* Flush non-transactional statistics */
+ flush_relation_anytime_stats(tabentry, &lstats->counts, false);
+
tabentry->tuples_inserted += lstats->counts.tuples_inserted;
tabentry->tuples_updated += lstats->counts.tuples_updated;
tabentry->tuples_deleted += lstats->counts.tuples_deleted;
@@ -866,9 +923,6 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
*/
tabentry->ins_since_vacuum += lstats->counts.tuples_inserted;
- tabentry->blocks_fetched += lstats->counts.blocks_fetched;
- tabentry->blocks_hit += lstats->counts.blocks_hit;
-
/* Clamp live_tuples in case of negative delta_live_tuples */
tabentry->live_tuples = Max(tabentry->live_tuples, 0);
/* Likewise for dead_tuples */
@@ -878,13 +932,11 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_o
/* The entry was successfully flushed, add the same to database stats */
dbentry = pgstat_prep_database_pending(dboid);
- dbentry->tuples_returned += lstats->counts.tuples_returned;
- dbentry->tuples_fetched += lstats->counts.tuples_fetched;
+ UPDATE_DATABASE_ANYTIME_STATS(dbentry, &lstats->counts);
+
dbentry->tuples_inserted += lstats->counts.tuples_inserted;
dbentry->tuples_updated += lstats->counts.tuples_updated;
dbentry->tuples_deleted += lstats->counts.tuples_deleted;
- dbentry->blocks_fetched += lstats->counts.blocks_fetched;
- dbentry->blocks_hit += lstats->counts.blocks_hit;
return true;
}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ef856dbf55b..06639198f28 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -21,6 +21,7 @@
#include "utils/backend_status.h" /* for backward compatibility */ /* IWYU pragma: export */
#include "utils/pgstat_kind.h"
#include "utils/relcache.h"
+#include "utils/timeout.h"
#include "utils/wait_event.h" /* for backward compatibility */ /* IWYU pragma: export */
@@ -537,10 +538,11 @@ extern void pgstat_report_anytime_stat(bool force);
extern void pgstat_force_next_flush(void);
/*
- * Schedule the next anytime stats update timeout.
+ * Schedule the next anytime stats update timeout and mark that we have
+ * mixed anytime stats pending.
*
* This should be called whenever accumulating statistics that support
- * FLUSH_ANYTIME flushing mode.
+ * FLUSH_ANYTIME or FLUSH_MIXED flushing modes.
*/
#define pgstat_schedule_anytime_update() \
do { \
@@ -703,37 +705,58 @@ extern void pgstat_report_analyze(Relation rel,
#define pgstat_count_heap_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_getnext(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_heap_fetch(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_scan(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.numscans++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_index_tuples(rel, n) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.tuples_returned += (n); \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_read(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_fetched++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
#define pgstat_count_buffer_hit(rel) \
do { \
if (pgstat_should_count_relation(rel)) \
+ { \
(rel)->pgstat_info->counts.blocks_hit++; \
+ pgstat_schedule_anytime_update(); \
+ } \
} while (0)
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out
index cfad309ccf3..11e3e57806d 100644
--- a/src/test/isolation/expected/stats.out
+++ b/src/test/isolation/expected/stats.out
@@ -2245,6 +2245,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/expected/stats_1.out b/src/test/isolation/expected/stats_1.out
index e1d937784cb..aef582e7582 100644
--- a/src/test/isolation/expected/stats_1.out
+++ b/src/test/isolation/expected/stats_1.out
@@ -2253,6 +2253,108 @@ seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum
(1 row)
+starting permutation: s2_begin s2_table_select s1_sleep s1_table_stats s2_track_counts_off s2_table_select s1_sleep s1_table_stats s2_track_counts_on s2_table_select s1_sleep s1_table_stats s2_table_drop s2_commit
+pg_stat_force_next_flush
+------------------------
+
+(1 row)
+
+step s2_begin: BEGIN;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_off: SET track_counts = off;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 1| 1| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_track_counts_on: SET track_counts = on;
+step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value;
+key|value
+---+-----
+k0 | 1
+(1 row)
+
+step s1_sleep: SELECT pg_sleep(1.5);
+pg_sleep
+--------
+
+(1 row)
+
+step s1_table_stats:
+ SELECT
+ pg_stat_get_numscans(tso.oid) AS seq_scan,
+ pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read,
+ pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins,
+ pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd,
+ pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del,
+ pg_stat_get_live_tuples(tso.oid) AS n_live_tup,
+ pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup,
+ pg_stat_get_vacuum_count(tso.oid) AS vacuum_count
+ FROM test_stat_oid AS tso
+ WHERE tso.name = 'test_stat_tab'
+
+seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count
+--------+------------+---------+---------+---------+----------+----------+------------
+ 2| 2| 1| 0| 0| 1| 0| 0
+(1 row)
+
+step s2_table_drop: DROP TABLE test_stat_tab;
+step s2_commit: COMMIT;
+
starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on
pg_stat_force_next_flush
------------------------
diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec
index da16710da0f..47414eb6009 100644
--- a/src/test/isolation/specs/stats.spec
+++ b/src/test/isolation/specs/stats.spec
@@ -50,6 +50,8 @@ step s1_rollback { ROLLBACK; }
step s1_prepare_a { PREPARE TRANSACTION 'a'; }
step s1_commit_prepared_a { COMMIT PREPARED 'a'; }
step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
+# Has to be greater than session 2 stats_flush_interval
+step s1_sleep { SELECT pg_sleep(1.5); }
# Function stats steps
step s1_ff { SELECT pg_stat_force_next_flush(); }
@@ -132,12 +134,16 @@ step s1_slru_check_stats {
session s2
-setup { SET stats_fetch_consistency = 'none'; }
+setup {
+ SET stats_fetch_consistency = 'none';
+ SET stats_flush_interval = '1s';
+}
step s2_begin { BEGIN; }
step s2_commit { COMMIT; }
step s2_commit_prepared_a { COMMIT PREPARED 'a'; }
step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; }
step s2_ff { SELECT pg_stat_force_next_flush(); }
+step s2_table_drop { DROP TABLE test_stat_tab; }
# Function stats steps
step s2_track_funcs_all { SET track_functions = 'all'; }
@@ -156,6 +162,8 @@ step s2_func_stats {
}
# Relation stats steps
+step s2_track_counts_on { SET track_counts = on; }
+step s2_track_counts_off { SET track_counts = off; }
step s2_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; }
step s2_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';}
@@ -435,6 +443,23 @@ permutation
s1_table_drop
s1_table_stats
+### Check that some stats are updated (seq_scan and seq_tup_read)
+### while the transaction is still running
+permutation
+ s2_begin
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_off
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_track_counts_on
+ s2_table_select
+ s1_sleep
+ s1_table_stats
+ s2_table_drop
+ s2_commit
### Check that we don't count changes with track counts off, but allow access
### to prior stats
--
2.34.1
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 16:45 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-28 11:14 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-31 17:16 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-02 17:16 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-02-04 16:19 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-04 16:26 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-02-06 11:09 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-02-14 02:23 ` Sami Imseih <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Sami Imseih @ 2026-02-14 02:23 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Michael Paquier <[email protected]>; Fujii Masao <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
> PFA attached v6, addressing the reviews comments.
Thanks for the patches!
v6 is getting closer IMO. Here are some comments I have.
v6-0001 looks solid, but some minor comments:
1/
+ pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true);
let's also use explicit (void) cast here
2/
+ * Timeout handler for flushing non-transactional stats.
I also noticed in v6-0005, we refer to "anytime" stats as
"non-transactional". It's better to just refer to them as "anytime"
anywhere instead of "non-transactional:.
v6-0002:
+ if (nowait && !LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE))
+ return true; /* failed to flush */
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
Here the LWLock acquisition will deadlock if we are nowait, successfully
acquire the lock conditionally, and then we try to acquire it again. The
logic here should not try to acquire the lock twice.
+ -- Force anytime flush (inside transaction!)
+ select pg_stat_force_anytime_flush();
Not sure why we need pg_stat_force_anytime_flush.
A pg_sleep is sufficient, like below. right?
```
select test_custom_stats_fixed_anytime_update() from generate_series(1, 2);
select pg_sleep(1.5);
select 'anytime:'||numcalls from test_custom_stats_fixed_report();
```
v6-0003:
1/
Suggested doc changes:
<para>
- Sets the interval at which statistics that can be updated while a
- transaction is still running are made visible. These include,
for example,
- WAL activity and I/O operations.
+ Sets the interval at which certain statistics, which can be
updated while a
+ transaction is in progress, are made visible. These include
WAL activity
+ and I/O operations.
Such statistics are refreshed at the specified interval and
can be observed
during active transactions in monitoring views such as
- <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>
+ <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>
and
- <link linkend="monitoring-pg-stat-wal-view"><structname>pg_stat_wal</structname></link>.
- Other statistics are only made visible at transaction end and are not
- affected by this setting.
+ <link linkend="monitoring-pg-stat-io-view"><structname>pg_stat_io</structname></link>.
If the value is specified without a unit, milliseconds are assumed.
The default is 10 seconds (<literal>10s</literal>), which is generally
the smallest practical value for long-running transactions.
- Other statistics are only made visible at transaction end and are not
- affected by this setting.
I removed this, because it's mentioned in the notes section later on.
2/
I don't see we have tests for other timeout based GUCs, but it would nice
to ensure that this woks correctly. Maybe as a custom_stats test where we
SET stats_flush_interval inside the transaction and make sure the stats flush
only after the new timeout has passed. Maybe?
v6-0004:
1/
NIT:
+$node_primary->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
+$node_publisher->append_conf('postgresql.conf', "stats_flush_interval= '1s'");
missing space before the equal sign.
v6-0005:
1/
/* Partial flush only happens in anytime mode for FLUSH_ANYTIME stats */
is_partial_flush = (anytime_only && kind_info->flush_mode ==
FLUSH_ANYTIME);
Will this be tue at all time? Let's imagine a Kind that flushes all the fields
ANYTIME, would we not want to delete the pending entry?
+ /* if successfull non-partial flush, remove entry */
+ if (did_flush && !is_partial_flush)
pgstat_delete_pending_entry(entry_ref);
2/ indentation:
Some statistics are updated while a transaction is in progress
(for example,
<structfield>blks_read</structfield>, <structfield>blks_hit</structfield>,
<structfield>tup_returned</structfield> and
<structfield>tup_fetched</structfield>).
- Statistics that either do not depend on transactions or require
transactional
- consistency are updated only when the transaction ends.
Statistics that require
- transactional consistency include <structfield>xact_commit</structfield>,
- <structfield>xact_rollback</structfield>,
<structfield>tup_inserted</structfield>,
- <structfield>tup_updated</structfield> and
<structfield>tup_deleted</structfield>.
+ Statistics that either do not depend on transactions or require
transactional
+ consistency are updated only when the transaction ends.
Statistics that require
+ transactional consistency include <structfield>xact_commit</structfield>,
+ <structfield>xact_rollback</structfield>,
<structfield>tup_inserted</structfield>,
+ <structfield>tup_updated</structfield> and
<structfield>tup_deleted</structfield>.
</para>
</note>
--
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 01:56 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
2026-01-22 07:43 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Re: Flush some statistics within running transactions Fujii Masao <[email protected]>
@ 2026-01-30 09:46 ` Álvaro Herrera <[email protected]>
1 sibling, 0 replies; 27+ messages in thread
From: Álvaro Herrera @ 2026-01-30 09:46 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Sami Imseih <[email protected]>; Michael Paquier <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
On 2026-Jan-22, Fujii Masao wrote:
> I haven't read the patch in detail yet, but after applying patch 0001 and
> causing a lock wait (for example, using the steps below), I observed that
> log_lock_waits messages are emitted every second.
Interesting. Bertrand asked me about this. He says[1] he diagnosed this
down to SetLatch being called immediately after the handler runs, and I
wonder if it's correct that other timeout handler functions used for
RegisterTimeout are doing SetLatch(). The SIGALRM handler
handle_sig_alarm() that calls the specific handler function already has
a SetLatch call, so why do we need another one here?
I tested this theory quickly by removing the SetLatch from
IdleStatsUpdateTimeoutHandler() and rerunning the tests. (This one
chosen because it has highest coverage per [2]). Everything passed,
though of course this is probably not proof enough.
Maybe we're just cargo-culting these SetLatch() calls?
[1] https://postgr.es/m/[email protected]
[2] https://coverage.postgresql.org/src/backend/utils/init/postinit.c.gcov.html
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Right now the sectors on the hard disk run clockwise, but I heard a rumor that
you can squeeze 0.2% more throughput by running them counterclockwise.
It's worth the effort. Recommended." (Gerry Pourwelle)
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
@ 2026-01-22 02:28 ` Michael Paquier <[email protected]>
2026-01-22 08:02 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
1 sibling, 1 reply; 27+ messages in thread
From: Michael Paquier @ 2026-01-22 02:28 UTC (permalink / raw)
To: Sami Imseih <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
On Wed, Jan 21, 2026 at 07:41:30PM -0600, Sami Imseih wrote:
> Another one would be n_mod_since_analyze, That should
> only be updated after commit (or not after rollback). Otherwise,
> it may throw autovanalyze threshold calculations way off. Same
> for n_dead_tup and autovacuum.
Point taken. It sounds like it is going to be super important to
document in the patch these kind of current expectations, so as one
does not flip the flush mode one way or another incorrectly, or
assigns an incorrect flush mode when adding a new stats kind. It's
probably worth documenting that the end-of-transaction flush should be
the default norm, while the out-of-transaction case should be an
exception one needs to be careful of.
> Sure, Bertrand mentioned early in the thread that the anytime flushes
> could be made configurable. Perhaps that is a good idea where we can
> default with something large like 10s intervals for anytime flushes, but allow
> the user to configure a more frequent flushes ( although I would think
> that 1 sec is the minimum we should allow ).
Sure, I am just mentioning that we should not be that aggressive for
everybody. If this can be made configurable on a call-basis, even if
it means a new GUC, that may be better in the long run.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, 2-signature.asc)
download
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-21 10:34 ` Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
2026-01-22 01:41 ` Re: Flush some statistics within running transactions Sami Imseih <[email protected]>
2026-01-22 02:28 ` Re: Flush some statistics within running transactions Michael Paquier <[email protected]>
@ 2026-01-22 08:02 ` Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Bertrand Drouvot @ 2026-01-22 08:02 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Sami Imseih <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>
Hi,
On Thu, Jan 22, 2026 at 11:28:06AM +0900, Michael Paquier wrote:
> On Wed, Jan 21, 2026 at 07:41:30PM -0600, Sami Imseih wrote:
> > Another one would be n_mod_since_analyze, That should
> > only be updated after commit (or not after rollback). Otherwise,
> > it may throw autovanalyze threshold calculations way off. Same
> > for n_dead_tup and autovacuum.
>
> Point taken. It sounds like it is going to be super important to
> document in the patch these kind of current expectations, so as one
> does not flip the flush mode one way or another incorrectly, or
> assigns an incorrect flush mode when adding a new stats kind. It's
> probably worth documenting that the end-of-transaction flush should be
> the default norm, while the out-of-transaction case should be an
> exception one needs to be careful of.
Agreed, I'll add more explanations around that.
> > Sure, Bertrand mentioned early in the thread that the anytime flushes
> > could be made configurable. Perhaps that is a good idea where we can
> > default with something large like 10s intervals for anytime flushes, but allow
> > the user to configure a more frequent flushes ( although I would think
> > that 1 sec is the minimum we should allow ).
>
> Sure, I am just mentioning that we should not be that aggressive for
> everybody.
I'm not opposed to increase the flush frequency but I suppose most of the monitoring
tools are sampling at a 1s frequency. So, if we set the flush frequency to say 10s,
that would result in "spikes" every 10s. That's misleading, because it's not a
spike in activity, it's a delay in reporting.
I think that would make sense if we expect the 1s interval to have a negative
impact, but that's not what I expect and observed.
> If this can be made configurable on a call-basis, even if
> it means a new GUC, that may be better in the long run.
If we think that the 1s interval is a problem, we could go in that direction.
Though it might be better to hardcode a larger value instead of letting the users
set values that could be problematic.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: Flush some statistics within running transactions
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
@ 2026-01-20 23:18 ` Zsolt Parragi <[email protected]>
1 sibling, 0 replies; 27+ messages in thread
From: Zsolt Parragi @ 2026-01-20 23:18 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: Sami Imseih <[email protected]>; [email protected]
Hello
@@ -264,6 +266,12 @@ typedef struct PgStat_KindInfo
/* Flush behavior */
PgStat_FlushBehavior flush_behavior;
+ /*
+ * For PGSTAT_FLUSH_MIXED kinds: callback to flush only some fields. If
+ * NULL for a MIXED kind, treated as PGSTAT_FLUSH_AT_TXN_BOUNDARY.
+ */
+ bool (*flush_anytime_cb) (PgStat_EntryRef *entry_ref, bool nowait);
+
The comment seems to use incorrect names, shouldn't be FLUSH_MIXED and
FLUSH_AT_TXN_BOUNDARY without PGSTAT_?
^ permalink raw reply [nested|flat] 27+ messages in thread
end of thread, other threads:[~2026-02-14 02:23 UTC | newest]
Thread overview: 27+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-01-19 12:17 Re: Flush some statistics within running transactions Bertrand Drouvot <[email protected]>
2026-01-20 19:27 ` Sami Imseih <[email protected]>
2026-01-21 10:34 ` Bertrand Drouvot <[email protected]>
2026-01-22 00:02 ` Michael Paquier <[email protected]>
2026-01-22 01:41 ` Sami Imseih <[email protected]>
2026-01-22 01:56 ` Fujii Masao <[email protected]>
2026-01-22 07:43 ` Bertrand Drouvot <[email protected]>
2026-01-22 12:12 ` Fujii Masao <[email protected]>
2026-01-22 16:45 ` Bertrand Drouvot <[email protected]>
2026-01-26 06:59 ` Bertrand Drouvot <[email protected]>
2026-01-28 06:35 ` Michael Paquier <[email protected]>
2026-01-28 11:14 ` Bertrand Drouvot <[email protected]>
2026-01-31 01:33 ` Sami Imseih <[email protected]>
2026-01-31 17:16 ` Sami Imseih <[email protected]>
2026-02-02 07:31 ` Michael Paquier <[email protected]>
2026-02-02 17:19 ` Bertrand Drouvot <[email protected]>
2026-02-02 17:16 ` Bertrand Drouvot <[email protected]>
2026-02-04 15:57 ` Zsolt Parragi <[email protected]>
2026-02-04 20:15 ` Sami Imseih <[email protected]>
2026-02-04 16:19 ` Sami Imseih <[email protected]>
2026-02-04 16:26 ` Sami Imseih <[email protected]>
2026-02-06 11:09 ` Bertrand Drouvot <[email protected]>
2026-02-14 02:23 ` Sami Imseih <[email protected]>
2026-01-30 09:46 ` Álvaro Herrera <[email protected]>
2026-01-22 02:28 ` Michael Paquier <[email protected]>
2026-01-22 08:02 ` Bertrand Drouvot <[email protected]>
2026-01-20 23:18 ` Zsolt Parragi <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox