pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Flush the IO statistics of active WAL senders more frequently 3+ messages / 1 participants [nested] [flat]
* pgsql: Flush the IO statistics of active WAL senders more frequently @ 2025-04-07 22:59 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 3+ messages in thread From: Michael Paquier @ 2025-04-07 22:59 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Flush the IO statistics of active WAL senders more frequently WAL senders do not flush their statistics until they exit, limiting the monitoring possible for live processes. This is penalizing when WAL senders are running for a long time, like in streaming or logical replication setups, because it is not possible to know the amount of IO they generate while running. This commit makes WAL senders more aggressive with their statistics flush, using an internal of 1 second, with the flush timing calculated based on the existing GetCurrentTimestamp() done before the sleeps done to wait for some activity. Note that the sleep done for logical and physical WAL senders happens in two different code paths, so the stats flushes need to happen in these two places. One test is added for the physical WAL sender case, and one for the logical WAL sender case. This can be done in a stable fashion by relying on the WAL generated by the TAP tests in combination with a stats reset while a server is running, but only on HEAD as WAL data has been added to pg_stat_io in a051e71e28a1. This issue exists since a9c70b46dbe and the introduction of pg_stat_io, so backpatch down to v16. Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> Reviewed-by: vignesh C <vignesh21@gmail.com> Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com> Discussion: https://postgr.es/m/Z73IsKBceoVd4t55@ip-10-97-1-34.eu-west-3.compute.internal Backpatch-through: 16 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/039549d70f6aa2daa3714a13752a08fa8ca2fb05 Modified Files -------------- src/backend/replication/walsender.c | 36 ++++++++++++++++++++++++++++-- src/test/recovery/t/001_stream_rep.pl | 16 +++++++++++++ src/test/subscription/t/001_rep_changes.pl | 16 +++++++++++++ 3 files changed, 66 insertions(+), 2 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
* pgsql: Flush the IO statistics of active WAL senders more frequently @ 2025-04-07 22:59 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 3+ messages in thread From: Michael Paquier @ 2025-04-07 22:59 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Flush the IO statistics of active WAL senders more frequently WAL senders do not flush their statistics until they exit, limiting the monitoring possible for live processes. This is penalizing when WAL senders are running for a long time, like in streaming or logical replication setups, because it is not possible to know the amount of IO they generate while running. This commit makes WAL senders more aggressive with their statistics flush, using an internal of 1 second, with the flush timing calculated based on the existing GetCurrentTimestamp() done before the sleeps done to wait for some activity. Note that the sleep done for logical and physical WAL senders happens in two different code paths, so the stats flushes need to happen in these two places. One test is added for the physical WAL sender case, and one for the logical WAL sender case. This can be done in a stable fashion by relying on the WAL generated by the TAP tests in combination with a stats reset while a server is running, but only on HEAD as WAL data has been added to pg_stat_io in a051e71e28a1. This issue exists since a9c70b46dbe and the introduction of pg_stat_io, so backpatch down to v16. Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> Reviewed-by: vignesh C <vignesh21@gmail.com> Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com> Discussion: https://postgr.es/m/Z73IsKBceoVd4t55@ip-10-97-1-34.eu-west-3.compute.internal Backpatch-through: 16 Branch ------ REL_17_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/5cbbe70a9cc6a9d8b7ed583cf66657be8ace0eaa Modified Files -------------- src/backend/replication/walsender.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
* pgsql: Flush the IO statistics of active WAL senders more frequently @ 2025-04-07 22:59 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 3+ messages in thread From: Michael Paquier @ 2025-04-07 22:59 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Flush the IO statistics of active WAL senders more frequently WAL senders do not flush their statistics until they exit, limiting the monitoring possible for live processes. This is penalizing when WAL senders are running for a long time, like in streaming or logical replication setups, because it is not possible to know the amount of IO they generate while running. This commit makes WAL senders more aggressive with their statistics flush, using an internal of 1 second, with the flush timing calculated based on the existing GetCurrentTimestamp() done before the sleeps done to wait for some activity. Note that the sleep done for logical and physical WAL senders happens in two different code paths, so the stats flushes need to happen in these two places. One test is added for the physical WAL sender case, and one for the logical WAL sender case. This can be done in a stable fashion by relying on the WAL generated by the TAP tests in combination with a stats reset while a server is running, but only on HEAD as WAL data has been added to pg_stat_io in a051e71e28a1. This issue exists since a9c70b46dbe and the introduction of pg_stat_io, so backpatch down to v16. Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> Reviewed-by: vignesh C <vignesh21@gmail.com> Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com> Discussion: https://postgr.es/m/Z73IsKBceoVd4t55@ip-10-97-1-34.eu-west-3.compute.internal Backpatch-through: 16 Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/e2a82cd23a88c7f10c9546fcc9fff94db25f66a0 Modified Files -------------- src/backend/replication/walsender.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2025-04-07 22:59 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-04-07 22:59 pgsql: Flush the IO statistics of active WAL senders more frequently Michael Paquier <michael@paquier.xyz> 2025-04-07 22:59 pgsql: Flush the IO statistics of active WAL senders more frequently Michael Paquier <michael@paquier.xyz> 2025-04-07 22:59 pgsql: Flush the IO statistics of active WAL senders more frequently Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox