From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers@lists.postgresql.org, Nazir Bilal Yavuz <byavuz81@gmail.com>
Subject: Re: per backend WAL statistics
Date: Mon, 17 Feb 2025 15:14:59 +0000
Message-ID: <Z7NSc5j6M4g2r1HD@ip-10-97-1-34.eu-west-3.compute.internal> (raw)
In-Reply-To: <Z7LsXq7AAnipA6BN@ip-10-97-1-34.eu-west-3.compute.internal>
References: <Z6HUpWt11kZNHey0@ip-10-97-1-34.eu-west-3.compute.internal>
<Z6LJ7xcXmFs_LKmV@paquier.xyz>
<Z6M7/zq4vgE4lch2@ip-10-97-1-34.eu-west-3.compute.internal>
<Z6M98Y5dTQoqb3kO@paquier.xyz>
<Z6N1eDP01ZFBcBrV@ip-10-97-1-34.eu-west-3.compute.internal>
<Z6QSr6rYB_fj9smy@paquier.xyz>
<Z6SO4FKamvH83/v2@ip-10-97-1-34.eu-west-3.compute.internal>
<Z7LeXGJOMRA+dCJM@ip-10-97-1-34.eu-west-3.compute.internal>
<Z7LkesxmaUg4lSVb@paquier.xyz>
<Z7LsXq7AAnipA6BN@ip-10-97-1-34.eu-west-3.compute.internal>
Hi,
On Mon, Feb 17, 2025 at 07:59:26AM +0000, Bertrand Drouvot wrote:
> Hi,
>
> On Mon, Feb 17, 2025 at 04:25:46PM +0900, Michael Paquier wrote:
> > On Mon, Feb 17, 2025 at 06:59:40AM +0000, Bertrand Drouvot wrote:
> > > There is still something that would simplify what is done here: it's the
> > > "the elimination of the write & sync columns for pg_stat_wal" mentioned in [2].
> >
> > Yeah, still you cannot just remove them because the data tracked in
> > pg_stat_io is not entirely the same, right?
>
> I think that we can just remove them. They are tracked and incremented at the
> exact same places in issue_xlog_fsync() and XLogWrite(). What differs is the
> "bytes" (as pg_stat_wal.wal_bytes somehow "focus" on the wal records size while
> the pg_stat_io's unit is the wal_block_size) and we keep them in both places.
> Also it looks like we can get rid of PendingWalStats...
PFA the whole picture. 0001 is implementing the fields removal in pg_stat_wal
(and also PendingWalStats). I think that's ok given the backend's type for which
pgstat_tracks_io_bktype() returns false. But now you make me doubt about 0001.
Anyway, it's probably better to move the 0001 discussion to a dedicated thread,
thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.comAttachments:download all (4 files, .tar.gz)v7-0001-Remove-wal_-sync-write-_time-from-pg_stat_wal.patch (text/x-diff, 17.7K) download
From 3e54a2f32b79238cefdd946e2fc017de0883c182 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 17 Feb 2025 07:17:49 +0000
Subject: [PATCH v7 1/4] Remove wal_[sync|write][_time] from pg_stat_wal
a051e71e28a added this information into pg_stat_io (with more details and
granularity), so there is no need to keep it in pg_stat_wal. This also
allows to remove PendingWalStats and simplifies up coming commits related
to per backend WAL statistics.
---
doc/src/sgml/config.sgml | 4 +-
doc/src/sgml/monitoring.sgml | 98 +++++++++++--------------
doc/src/sgml/wal.sgml | 9 ++-
src/backend/access/transam/xlog.c | 27 -------
src/backend/catalog/system_views.sql | 4 -
src/backend/utils/activity/pgstat_wal.c | 24 +-----
src/backend/utils/adt/pgstatfuncs.c | 20 +----
src/include/catalog/pg_proc.dat | 6 +-
src/include/pgstat.h | 28 -------
src/test/regress/expected/rules.out | 6 +-
src/tools/pgindent/typedefs.list | 1 -
11 files changed, 57 insertions(+), 170 deletions(-)
56.1% doc/src/sgml/
6.6% src/backend/access/transam/
11.6% src/backend/utils/activity/
11.0% src/backend/utils/adt/
3.4% src/include/catalog/
7.7% src/include/
3.3% src/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgmlindex 336630ce417..9fff9048551 100644--- a/doc/src/sgml/config.sgml+++ b/doc/src/sgml/config.sgml@@ -8341,8 +8341,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
You can use the <application>pg_test_timing</application> tool to
measure the overhead of timing on your system.
I/O timing information is
- displayed in <link linkend="monitoring-pg-stat-wal-view">- <structname>pg_stat_wal</structname></link>.+ displayed in <link linkend="monitoring-pg-stat-io-view">+ <structname>pg_stat_io</structname></link> for the wal <literal>object</literal>.
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this setting.
</para>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgmlindex 928a6eb64b0..e645591ce53 100644--- a/doc/src/sgml/monitoring.sgml+++ b/doc/src/sgml/monitoring.sgml@@ -2930,6 +2930,47 @@ description | Waiting for a newly initialized WAL file to reach durable storage
writer</literal>.
</para>
+ <para>+ For the <literal>wal</literal> <structfield>object</structfield>:+ <itemizedlist>+ <listitem>+ <para>+ <structfield>writes</structfield> is the number of times WAL buffers were+ written out to disk via <function>XLogWrite</function> request (See <xref linkend="wal-configuration"/>+ for more information about the internal WAL function <function>XLogWrite</function>).+ </para>+ </listitem>+ <listitem>+ <para>+ <structfield>fsyncs</structfield> is the number of times WAL files were+ synced to disk via <function>issue_xlog_fsync</function> request (if <xref linkend="guc-fsync"/>+ is <literal>on</literal> and <xref linkend="guc-wal-sync-method"/> is either+ <literal>fdatasync</literal>, <literal>fsync</literal> or <literal>fsync_writethrough</literal>,+ otherwise zero) (See <xref linkend="wal-configuration"/> for more information about+ the internal WAL function <function>issue_xlog_fsync</function>).+ </para>+ </listitem>+ <listitem>+ <para>+ <structfield>write_time</structfield> is the total amount of time spent writing+ WAL buffers to disk via <function>XLogWrite</function> request (if <xref linkend="guc-track-wal-io-timing"/>+ is enabled, otherwise zero). This includes the sync time when <varname>wal_sync_method</varname>+ is either <literal>open_datasync</literal> or <literal>open_sync</literal>.+ </para>+ </listitem>+ <listitem>+ <para>+ <structfield>sync_time</structfield> is the total amount of time spent syncing+ WAL files to disk via <function>issue_xlog_fsync</function> request (if+ <varname>track_wal_io_timing</varname> is enabled, <varname>fsync</varname> is+ <literal>on</literal>, and <varname>wal_sync_method</varname> is either+ <literal>fdatasync</literal>, <literal>fsync</literal> or <literal>fsync_writethrough</literal>,+ otherwise zero).+ </para>+ </listitem>+ </itemizedlist>+ </para>+
<para>
<structname>pg_stat_io</structname> can be used to inform database tuning.
For example:
@@ -3255,63 +3296,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para></entry>
</row>
- <row>- <entry role="catalog_table_entry"><para role="column_definition">- <structfield>wal_write</structfield> <type>bigint</type>- </para>- <para>- Number of times WAL buffers were written out to disk via- <function>XLogWrite</function> request.- See <xref linkend="wal-configuration"/> for more information about- the internal WAL function <function>XLogWrite</function>.- </para></entry>- </row>-- <row>- <entry role="catalog_table_entry"><para role="column_definition">- <structfield>wal_sync</structfield> <type>bigint</type>- </para>- <para>- Number of times WAL files were synced to disk via- <function>issue_xlog_fsync</function> request- (if <xref linkend="guc-fsync"/> is <literal>on</literal> and- <xref linkend="guc-wal-sync-method"/> is either- <literal>fdatasync</literal>, <literal>fsync</literal> or- <literal>fsync_writethrough</literal>, otherwise zero).- See <xref linkend="wal-configuration"/> for more information about- the internal WAL function <function>issue_xlog_fsync</function>.- </para></entry>- </row>-- <row>- <entry role="catalog_table_entry"><para role="column_definition">- <structfield>wal_write_time</structfield> <type>double precision</type>- </para>- <para>- Total amount of time spent writing WAL buffers to disk via- <function>XLogWrite</function> request, in milliseconds- (if <xref linkend="guc-track-wal-io-timing"/> is enabled,- otherwise zero). This includes the sync time when- <varname>wal_sync_method</varname> is either- <literal>open_datasync</literal> or <literal>open_sync</literal>.- </para></entry>- </row>-- <row>- <entry role="catalog_table_entry"><para role="column_definition">- <structfield>wal_sync_time</structfield> <type>double precision</type>- </para>- <para>- Total amount of time spent syncing WAL files to disk via- <function>issue_xlog_fsync</function> request, in milliseconds- (if <varname>track_wal_io_timing</varname> is enabled,- <varname>fsync</varname> is <literal>on</literal>, and- <varname>wal_sync_method</varname> is either- <literal>fdatasync</literal>, <literal>fsync</literal> or- <literal>fsync_writethrough</literal>, otherwise zero).- </para></entry>- </row>-
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>stats_reset</structfield> <type>timestamp with time zone</type>
diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgmlindex b908720adea..bb4892413fc 100644--- a/doc/src/sgml/wal.sgml+++ b/doc/src/sgml/wal.sgml@@ -813,8 +813,8 @@
When <xref linkend="guc-track-wal-io-timing"/> is enabled, the total
amounts of time <function>XLogWrite</function> writes and
<function>issue_xlog_fsync</function> syncs WAL data to disk are counted as
- <literal>wal_write_time</literal> and <literal>wal_sync_time</literal> in- <xref linkend="pg-stat-wal-view"/>, respectively.+ <literal>write_time</literal> and <literal>sync_time</literal> in+ <xref linkend="pg-stat-io-view"/> for the wal <literal>object</literal>, respectively.
<function>XLogWrite</function> is normally called by
<function>XLogInsertRecord</function> (when there is no space for the new
record in WAL buffers), <function>XLogFlush</function> and the WAL writer,
@@ -832,8 +832,9 @@
of the setting of <varname>track_wal_io_timing</varname>, the number
of times <function>XLogWrite</function> writes and
<function>issue_xlog_fsync</function> syncs WAL data to disk are also
- counted as <literal>wal_write</literal> and <literal>wal_sync</literal>- in <structname>pg_stat_wal</structname>, respectively.+ counted as <literal>write</literal> and <literal>sync</literal>+ in <structname>pg_stat_io</structname> for the <literal>wal</literal> object+ respectively.
</para>
<para>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.cindex 25a5c605404..020b99d402c 100644--- a/src/backend/access/transam/xlog.c+++ b/src/backend/access/transam/xlog.c@@ -2448,20 +2448,6 @@ XLogWrite(XLogwrtRqst WriteRqst, TimeLineID tli, bool flexible)
pgstat_count_io_op_time(IOOBJECT_WAL, IOCONTEXT_NORMAL,
IOOP_WRITE, start, 1, written);
- /*- * Increment the I/O timing and the number of times WAL data- * were written out to disk.- */- if (track_wal_io_timing)- {- instr_time end;-- INSTR_TIME_SET_CURRENT(end);- INSTR_TIME_ACCUM_DIFF(PendingWalStats.wal_write_time, end, start);- }-- PendingWalStats.wal_write++;-
if (written <= 0)
{
char xlogfname[MAXFNAMELEN];
@@ -8767,21 +8753,8 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
pgstat_report_wait_end();
- /*- * Increment the I/O timing and the number of times WAL files were synced.- */- if (track_wal_io_timing)- {- instr_time end;-- INSTR_TIME_SET_CURRENT(end);- INSTR_TIME_ACCUM_DIFF(PendingWalStats.wal_sync_time, end, start);- }-
pgstat_count_io_op_time(IOOBJECT_WAL, IOCONTEXT_NORMAL, IOOP_FSYNC,
start, 1, 0);
-- PendingWalStats.wal_sync++;
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sqlindex eff0990957e..a4d2cfdcaf5 100644--- a/src/backend/catalog/system_views.sql+++ b/src/backend/catalog/system_views.sql@@ -1189,10 +1189,6 @@ CREATE VIEW pg_stat_wal AS
w.wal_fpi,
w.wal_bytes,
w.wal_buffers_full,
- w.wal_write,- w.wal_sync,- w.wal_write_time,- w.wal_sync_time,
w.stats_reset
FROM pg_stat_get_wal() w;
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.cindex c1ca65eb991..4dc41a4a590 100644--- a/src/backend/utils/activity/pgstat_wal.c+++ b/src/backend/utils/activity/pgstat_wal.c@@ -21,8 +21,6 @@
#include "utils/pgstat_internal.h"
-PgStat_PendingWalStats PendingWalStats = {0};-
/*
* WAL usage counters saved from pgWalUsage at the previous call to
* pgstat_report_wal(). This is used to calculate how much WAL usage
@@ -118,17 +116,10 @@ pgstat_wal_flush_cb(bool nowait)
#define WALSTAT_ACC(fld, var_to_add) \
(stats_shmem->stats.fld += var_to_add.fld)
-#define WALSTAT_ACC_INSTR_TIME(fld) \- (stats_shmem->stats.fld += INSTR_TIME_GET_MICROSEC(PendingWalStats.fld))
WALSTAT_ACC(wal_records, wal_usage_diff);
WALSTAT_ACC(wal_fpi, wal_usage_diff);
WALSTAT_ACC(wal_bytes, wal_usage_diff);
WALSTAT_ACC(wal_buffers_full, wal_usage_diff);
- WALSTAT_ACC(wal_write, PendingWalStats);- WALSTAT_ACC(wal_sync, PendingWalStats);- WALSTAT_ACC_INSTR_TIME(wal_write_time);- WALSTAT_ACC_INSTR_TIME(wal_sync_time);-#undef WALSTAT_ACC_INSTR_TIME
#undef WALSTAT_ACC
LWLockRelease(&stats_shmem->lock);
@@ -138,11 +129,6 @@ pgstat_wal_flush_cb(bool nowait)
*/
prevWalUsage = pgWalUsage;
- /*- * Clear out the statistics buffer, so it can be re-used.- */- MemSet(&PendingWalStats, 0, sizeof(PendingWalStats));-
return false;
}
@@ -158,18 +144,12 @@ pgstat_wal_init_backend_cb(void)
}
/*
- * To determine whether any WAL activity has occurred since last time, not- * only the number of generated WAL records but also the numbers of WAL- * writes and syncs need to be checked. Because even transaction that- * generates no WAL records can write or sync WAL data when flushing the- * data pages.+ * To determine whether WAL usage happened.
*/
bool
pgstat_wal_have_pending_cb(void)
{
- return pgWalUsage.wal_records != prevWalUsage.wal_records ||- PendingWalStats.wal_write != 0 ||- PendingWalStats.wal_sync != 0;+ return pgWalUsage.wal_records != prevWalUsage.wal_records;
}
void
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.cindex e9096a88492..68e16e52ab6 100644--- a/src/backend/utils/adt/pgstatfuncs.c+++ b/src/backend/utils/adt/pgstatfuncs.c@@ -1637,7 +1637,7 @@ pg_stat_get_backend_io(PG_FUNCTION_ARGS)
Datum
pg_stat_get_wal(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_COLS 9+#define PG_STAT_GET_WAL_COLS 5
TupleDesc tupdesc;
Datum values[PG_STAT_GET_WAL_COLS] = {0};
bool nulls[PG_STAT_GET_WAL_COLS] = {0};
@@ -1654,15 +1654,7 @@ pg_stat_get_wal(PG_FUNCTION_ARGS)
NUMERICOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 4, "wal_buffers_full",
INT8OID, -1, 0);
- TupleDescInitEntry(tupdesc, (AttrNumber) 5, "wal_write",- INT8OID, -1, 0);- TupleDescInitEntry(tupdesc, (AttrNumber) 6, "wal_sync",- INT8OID, -1, 0);- TupleDescInitEntry(tupdesc, (AttrNumber) 7, "wal_write_time",- FLOAT8OID, -1, 0);- TupleDescInitEntry(tupdesc, (AttrNumber) 8, "wal_sync_time",- FLOAT8OID, -1, 0);- TupleDescInitEntry(tupdesc, (AttrNumber) 9, "stats_reset",+ TupleDescInitEntry(tupdesc, (AttrNumber) 5, "stats_reset",
TIMESTAMPTZOID, -1, 0);
BlessTupleDesc(tupdesc);
@@ -1682,14 +1674,8 @@ pg_stat_get_wal(PG_FUNCTION_ARGS)
Int32GetDatum(-1));
values[3] = Int64GetDatum(wal_stats->wal_buffers_full);
- values[4] = Int64GetDatum(wal_stats->wal_write);- values[5] = Int64GetDatum(wal_stats->wal_sync);-- /* Convert counters from microsec to millisec for display */- values[6] = Float8GetDatum(((double) wal_stats->wal_write_time) / 1000.0);- values[7] = Float8GetDatum(((double) wal_stats->wal_sync_time) / 1000.0);- values[8] = TimestampTzGetDatum(wal_stats->stat_reset_timestamp);+ values[4] = TimestampTzGetDatum(wal_stats->stat_reset_timestamp);
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 9e803d610d7..1e1626964e3 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -5950,9 +5950,9 @@
{ oid => '1136', descr => 'statistics: information about WAL activity',
proname => 'pg_stat_get_wal', proisstrict => 'f', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int8,int8,numeric,int8,int8,int8,float8,float8,timestamptz}',- proargmodes => '{o,o,o,o,o,o,o,o,o}',- proargnames => '{wal_records,wal_fpi,wal_bytes,wal_buffers_full,wal_write,wal_sync,wal_write_time,wal_sync_time,stats_reset}',+ proallargtypes => '{int8,int8,numeric,int8,timestamptz}',+ proargmodes => '{o,o,o,o,o}',+ proargnames => '{wal_records,wal_fpi,wal_bytes,wal_buffers_full,stats_reset}',
prosrc => 'pg_stat_get_wal' },
{ oid => '6248', descr => 'statistics: information about WAL prefetching',
proname => 'pg_stat_get_recovery_prefetch', prorows => '1', proretset => 't',
diff --git a/src/include/pgstat.h b/src/include/pgstat.hindex 53f2a8458e6..a3a341cc604 100644--- a/src/include/pgstat.h+++ b/src/include/pgstat.h@@ -480,28 +480,9 @@ typedef struct PgStat_WalStats
PgStat_Counter wal_fpi;
uint64 wal_bytes;
PgStat_Counter wal_buffers_full;
- PgStat_Counter wal_write;- PgStat_Counter wal_sync;- PgStat_Counter wal_write_time;- PgStat_Counter wal_sync_time;
TimestampTz stat_reset_timestamp;
} PgStat_WalStats;
-/*- * This struct stores wal-related durations as instr_time, which makes it- * cheaper and easier to accumulate them, by not requiring type- * conversions. During stats flush instr_time will be converted into- * microseconds.- */-typedef struct PgStat_PendingWalStats-{- PgStat_Counter wal_write;- PgStat_Counter wal_sync;- instr_time wal_write_time;- instr_time wal_sync_time;-} PgStat_PendingWalStats;--
/*
* Functions in pgstat.c
*/
@@ -834,13 +815,4 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime;
/* updated by the traffic cop and in errfinish() */
extern PGDLLIMPORT SessionEndType pgStatSessionEndCause;
--/*- * Variables in pgstat_wal.c- */--/* updated directly by backends and background processes */-extern PGDLLIMPORT PgStat_PendingWalStats PendingWalStats;--
#endif /* PGSTAT_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.outindex 5baba8d39ff..62f69ac20b2 100644--- a/src/test/regress/expected/rules.out+++ b/src/test/regress/expected/rules.out@@ -2259,12 +2259,8 @@ pg_stat_wal| SELECT wal_records,
wal_fpi,
wal_bytes,
wal_buffers_full,
- wal_write,- wal_sync,- wal_write_time,- wal_sync_time,
stats_reset
- FROM pg_stat_get_wal() w(wal_records, wal_fpi, wal_bytes, wal_buffers_full, wal_write, wal_sync, wal_write_time, wal_sync_time, stats_reset);+ FROM pg_stat_get_wal() w(wal_records, wal_fpi, wal_bytes, wal_buffers_full, stats_reset);
pg_stat_wal_receiver| SELECT pid,
status,
receive_start_lsn,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex bce4214503d..740777127e9 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -2163,7 +2163,6 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
-PgStat_PendingWalStats
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
v7-0002-Extract-logic-filling-pg_stat_get_wal-s-tuple-int.patch (text/x-diff, 3.3K) download
From c516da1bd0730c274d892999f8c390138dd3c8aa Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 6 Jan 2025 07:51:27 +0000
Subject: [PATCH v7 2/4] Extract logic filling pg_stat_get_wal()'s tuple into
its own routine
This commit adds pg_stat_wal_build_tuple(), a helper routine for
pg_stat_get_wal(), that fills its tuple based on the contents
of PgStat_WalStats. This will be used in a follow-up commit that uses
the same structures as pg_stat_wal for reporting, but for the PGSTAT_KIND_BACKEND
statistics kind.
---
src/backend/utils/adt/pgstatfuncs.c | 48 +++++++++++++++++++----------
1 file changed, 32 insertions(+), 16 deletions(-)
100.0% src/backend/utils/adt/
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.cindex 68e16e52ab6..620d60a0938 100644--- a/src/backend/utils/adt/pgstatfuncs.c+++ b/src/backend/utils/adt/pgstatfuncs.c@@ -1632,20 +1632,22 @@ pg_stat_get_backend_io(PG_FUNCTION_ARGS)
}
/*
- * Returns statistics of WAL activity+ * pg_stat_wal_build_tuple+ *+ * Helper routine for pg_stat_get_wal() returning one tuple based on the contents+ * of wal_stats.
*/
-Datum-pg_stat_get_wal(PG_FUNCTION_ARGS)+static Datum+pg_stat_wal_build_tuple(PgStat_WalStats wal_stats)
{
-#define PG_STAT_GET_WAL_COLS 5+#define PG_STAT_WAL_COLS 5
TupleDesc tupdesc;
- Datum values[PG_STAT_GET_WAL_COLS] = {0};- bool nulls[PG_STAT_GET_WAL_COLS] = {0};+ Datum values[PG_STAT_WAL_COLS] = {0};+ bool nulls[PG_STAT_WAL_COLS] = {0};
char buf[256];
- PgStat_WalStats *wal_stats;
/* Initialise attributes information in the tuple descriptor */
- tupdesc = CreateTemplateTupleDesc(PG_STAT_GET_WAL_COLS);+ tupdesc = CreateTemplateTupleDesc(PG_STAT_WAL_COLS);
TupleDescInitEntry(tupdesc, (AttrNumber) 1, "wal_records",
INT8OID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2, "wal_fpi",
@@ -1659,28 +1661,42 @@ pg_stat_get_wal(PG_FUNCTION_ARGS)
BlessTupleDesc(tupdesc);
- /* Get statistics about WAL activity */- wal_stats = pgstat_fetch_stat_wal();-
/* Fill values and NULLs */
- values[0] = Int64GetDatum(wal_stats->wal_records);- values[1] = Int64GetDatum(wal_stats->wal_fpi);+ values[0] = Int64GetDatum(wal_stats.wal_records);+ values[1] = Int64GetDatum(wal_stats.wal_fpi);
/* Convert to numeric. */
- snprintf(buf, sizeof buf, UINT64_FORMAT, wal_stats->wal_bytes);+ snprintf(buf, sizeof buf, UINT64_FORMAT, wal_stats.wal_bytes);
values[2] = DirectFunctionCall3(numeric_in,
CStringGetDatum(buf),
ObjectIdGetDatum(0),
Int32GetDatum(-1));
- values[3] = Int64GetDatum(wal_stats->wal_buffers_full);+ values[3] = Int64GetDatum(wal_stats.wal_buffers_full);- values[4] = TimestampTzGetDatum(wal_stats->stat_reset_timestamp);+ if (wal_stats.stat_reset_timestamp != 0)+ values[4] = TimestampTzGetDatum(wal_stats.stat_reset_timestamp);+ else+ nulls[4] = true;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+/*+ * Returns statistics of WAL activity+ */+Datum+pg_stat_get_wal(PG_FUNCTION_ARGS)+{+ PgStat_WalStats *wal_stats;++ /* Get statistics about WAL activity */+ wal_stats = pgstat_fetch_stat_wal();++ return (pg_stat_wal_build_tuple(*wal_stats));+}+
/*
* Returns statistics of SLRU caches.
*/
--
2.34.1
v7-0003-Adding-a-new-PgStat_WalCounters-struct.patch (text/x-diff, 4.2K) download
From a33621c2a2468c9e908622ed9940d15b19c2d843 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 16 Jan 2025 15:06:01 +0000
Subject: [PATCH v7 3/4] Adding a new PgStat_WalCounters struct
This new struct contains only the counters related to the WAL statistics.
This will be used in a follow-up commit that uses the same structures but
for the PGSTAT_KIND_BACKEND statistics kind.
---
src/backend/utils/activity/pgstat_wal.c | 2 +-
src/backend/utils/adt/pgstatfuncs.c | 20 +++++++++++---------
src/include/pgstat.h | 7 ++++++-
src/tools/pgindent/typedefs.list | 1 +
4 files changed, 19 insertions(+), 11 deletions(-)
8.5% src/backend/utils/activity/
82.0% src/backend/utils/adt/
7.8% src/include/
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.cindex 4dc41a4a590..6d024872701 100644--- a/src/backend/utils/activity/pgstat_wal.c+++ b/src/backend/utils/activity/pgstat_wal.c@@ -115,7 +115,7 @@ pgstat_wal_flush_cb(bool nowait)
return true;
#define WALSTAT_ACC(fld, var_to_add) \
- (stats_shmem->stats.fld += var_to_add.fld)+ (stats_shmem->stats.wal_counters.fld += var_to_add.fld)
WALSTAT_ACC(wal_records, wal_usage_diff);
WALSTAT_ACC(wal_fpi, wal_usage_diff);
WALSTAT_ACC(wal_bytes, wal_usage_diff);
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.cindex 620d60a0938..9de14ffd449 100644--- a/src/backend/utils/adt/pgstatfuncs.c+++ b/src/backend/utils/adt/pgstatfuncs.c@@ -1635,10 +1635,11 @@ pg_stat_get_backend_io(PG_FUNCTION_ARGS)
* pg_stat_wal_build_tuple
*
* Helper routine for pg_stat_get_wal() returning one tuple based on the contents
- * of wal_stats.+ * of wal_counters.
*/
static Datum
-pg_stat_wal_build_tuple(PgStat_WalStats wal_stats)+pg_stat_wal_build_tuple(PgStat_WalCounters wal_counters,+ TimestampTz stat_reset_timestamp)
{
#define PG_STAT_WAL_COLS 5
TupleDesc tupdesc;
@@ -1662,20 +1663,20 @@ pg_stat_wal_build_tuple(PgStat_WalStats wal_stats)
BlessTupleDesc(tupdesc);
/* Fill values and NULLs */
- values[0] = Int64GetDatum(wal_stats.wal_records);- values[1] = Int64GetDatum(wal_stats.wal_fpi);+ values[0] = Int64GetDatum(wal_counters.wal_records);+ values[1] = Int64GetDatum(wal_counters.wal_fpi);
/* Convert to numeric. */
- snprintf(buf, sizeof buf, UINT64_FORMAT, wal_stats.wal_bytes);+ snprintf(buf, sizeof buf, UINT64_FORMAT, wal_counters.wal_bytes);
values[2] = DirectFunctionCall3(numeric_in,
CStringGetDatum(buf),
ObjectIdGetDatum(0),
Int32GetDatum(-1));
- values[3] = Int64GetDatum(wal_stats.wal_buffers_full);+ values[3] = Int64GetDatum(wal_counters.wal_buffers_full);- if (wal_stats.stat_reset_timestamp != 0)- values[4] = TimestampTzGetDatum(wal_stats.stat_reset_timestamp);+ if (stat_reset_timestamp != 0)+ values[4] = TimestampTzGetDatum(stat_reset_timestamp);
else
nulls[4] = true;
@@ -1694,7 +1695,8 @@ pg_stat_get_wal(PG_FUNCTION_ARGS)
/* Get statistics about WAL activity */
wal_stats = pgstat_fetch_stat_wal();
- return (pg_stat_wal_build_tuple(*wal_stats));+ return (pg_stat_wal_build_tuple(wal_stats->wal_counters,+ wal_stats->stat_reset_timestamp));
}
/*
diff --git a/src/include/pgstat.h b/src/include/pgstat.hindex a3a341cc604..459a7cb328e 100644--- a/src/include/pgstat.h+++ b/src/include/pgstat.h@@ -474,12 +474,17 @@ typedef struct PgStat_StatTabEntry
PgStat_Counter total_autoanalyze_time;
} PgStat_StatTabEntry;
-typedef struct PgStat_WalStats+typedef struct PgStat_WalCounters
{
PgStat_Counter wal_records;
PgStat_Counter wal_fpi;
uint64 wal_bytes;
PgStat_Counter wal_buffers_full;
+} PgStat_WalCounters;++typedef struct PgStat_WalStats+{+ PgStat_WalCounters wal_counters;
TimestampTz stat_reset_timestamp;
} PgStat_WalStats;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex 740777127e9..77b47a6df7e 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -2178,6 +2178,7 @@ PgStat_SubXactStatus
PgStat_TableCounts
PgStat_TableStatus
PgStat_TableXactStatus
+PgStat_WalCounters
PgStat_WalStats
PgXmlErrorContext
PgXmlStrictness
--
2.34.1
v7-0004-per-backend-WAL-statistics.patch (text/x-diff, 13.0K) download
From 7665e01b6cf0b6bae3d277ce150387aff15e8f0f Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 6 Jan 2025 10:00:00 +0000
Subject: [PATCH v7 4/4] per backend WAL statistics
Now that commit 9aea73fc61 added backend-level statistics to pgstats (and
per backend IO statistics) we can more easily add per backend statistics.
This commit adds per backend WAL statistics using the same layer as pg_stat_wal,
except that it is now possible to know how much WAL activity is happening in each
backend rather than an overall aggregate of all the activity. A function called
pg_stat_get_backend_wal() is added to access this data depending on the
PID of a backend.
The same limitation as in 9aea73fc61 persists, meaning that Auxiliary processes
are not included in this set of statistics.
XXX: bump catalog version
---
doc/src/sgml/monitoring.sgml | 19 ++++++
src/backend/utils/activity/pgstat_backend.c | 64 +++++++++++++++++++++
src/backend/utils/activity/pgstat_wal.c | 2 +
src/backend/utils/adt/pgstatfuncs.c | 52 ++++++++++++++++-
src/include/catalog/pg_proc.dat | 7 +++
src/include/pgstat.h | 13 +++--
src/include/utils/pgstat_internal.h | 3 +-
src/test/regress/expected/stats.out | 14 +++++
src/test/regress/sql/stats.sql | 6 ++
9 files changed, 171 insertions(+), 9 deletions(-)
14.0% doc/src/sgml/
34.5% src/backend/utils/activity/
25.9% src/backend/utils/adt/
7.7% src/include/catalog/
3.9% src/include/utils/
7.4% src/test/regress/expected/
5.6% src/test/regress/sql/
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgmlindex e645591ce53..bd3fc5bd4c6 100644--- a/doc/src/sgml/monitoring.sgml+++ b/doc/src/sgml/monitoring.sgml@@ -4859,6 +4859,25 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para></entry>
</row>
+ <row>+ <entry id="pg-stat-get-backend-wal" role="func_table_entry"><para role="func_signature">+ <indexterm>+ <primary>pg_stat_get_backend_wal</primary>+ </indexterm>+ <function>pg_stat_get_backend_wal</function> ( <type>integer</type> )+ <returnvalue>record</returnvalue>+ </para>+ <para>+ Returns WAL statistics about the backend with the specified+ process ID. The output fields are exactly the same as the ones in the+ <structname>pg_stat_wal</structname> view.+ </para>+ <para>+ The function does not return WAL statistics for the checkpointer,+ the background writer, the startup process and the autovacuum launcher.+ </para></entry>+ </row>+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.cindex 4a667e7019c..cf3fac678c2 100644--- a/src/backend/utils/activity/pgstat_backend.c+++ b/src/backend/utils/activity/pgstat_backend.c@@ -35,6 +35,14 @@
*/
static PgStat_BackendPending PendingBackendStats;
+/*+ * WAL usage counters saved from pgWalUsage at the previous call to+ * pgstat_report_wal(). This is used to calculate how much WAL usage+ * happens between pgstat_report_wal() calls, by subtracting+ * the previous counters from the current ones.+ */+static WalUsage prevBackendWalUsage;+
/*
* Utility routines to report I/O stats for backends, kept here to avoid
* exposing PendingBackendStats to the outside world.
@@ -131,6 +139,57 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref)
MemSet(&PendingBackendStats.pending_io, 0, sizeof(PgStat_PendingIO));
}
+/*+ * To determine whether WAL usage happened.+ */+static bool+pgstat_backend_wal_have_pending(void)+{+ return pgWalUsage.wal_records != prevBackendWalUsage.wal_records;+}++/*+ * Flush out locally pending backend WAL statistics. Locking is managed+ * by the caller.+ */+static void+pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref)+{+ PgStatShared_Backend *shbackendent;+ PgStat_WalCounters *bktype_shstats;+ WalUsage wal_usage_diff = {0};++ /*+ * This function can be called even if nothing at all has happened. Avoid+ * taking lock for nothing in that case.+ */+ if (!pgstat_backend_wal_have_pending())+ return;++ shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats;+ bktype_shstats = &shbackendent->stats.wal_counters;++ /*+ * We don't update the WAL usage portion of the local WalStats elsewhere.+ * Calculate how much WAL usage counters were increased by subtracting the+ * previous counters from the current ones.+ */+ WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevBackendWalUsage);++#define WALSTAT_ACC(fld, var_to_add) \+ (bktype_shstats->fld += var_to_add.fld)+ WALSTAT_ACC(wal_buffers_full, wal_usage_diff);+ WALSTAT_ACC(wal_records, wal_usage_diff);+ WALSTAT_ACC(wal_fpi, wal_usage_diff);+ WALSTAT_ACC(wal_bytes, wal_usage_diff);+#undef WALSTAT_ACC++ /*+ * Save the current counters for the subsequent calculation of WAL usage.+ */+ prevBackendWalUsage = pgWalUsage;+}+
/*
* Flush out locally pending backend statistics
*
@@ -158,6 +217,9 @@ pgstat_flush_backend(bool nowait, bits32 flags)
if (flags & PGSTAT_BACKEND_FLUSH_IO)
pgstat_flush_backend_entry_io(entry_ref);
+ if (flags & PGSTAT_BACKEND_FLUSH_WAL)+ pgstat_flush_backend_entry_wal(entry_ref);+
pgstat_unlock_entry(entry_ref);
return false;
@@ -205,6 +267,8 @@ pgstat_create_backend(ProcNumber procnum)
pgstat_unlock_entry(entry_ref);
MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending));
++ prevBackendWalUsage = pgWalUsage;
}
/*
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.cindex 6d024872701..f268a610fb8 100644--- a/src/backend/utils/activity/pgstat_wal.c+++ b/src/backend/utils/activity/pgstat_wal.c@@ -53,6 +53,8 @@ pgstat_report_wal(bool force)
/* flush wal stats */
pgstat_flush_wal(nowait);
+ pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL);+
/* flush IO stats */
pgstat_flush_io(nowait);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.cindex 9de14ffd449..6676245651c 100644--- a/src/backend/utils/adt/pgstatfuncs.c+++ b/src/backend/utils/adt/pgstatfuncs.c@@ -1634,8 +1634,8 @@ pg_stat_get_backend_io(PG_FUNCTION_ARGS)
/*
* pg_stat_wal_build_tuple
*
- * Helper routine for pg_stat_get_wal() returning one tuple based on the contents- * of wal_counters.+ * Helper routine for pg_stat_get_wal() and pg_stat_get_backend_wal() returning+ * one tuple based on the contents of wal_counters.
*/
static Datum
pg_stat_wal_build_tuple(PgStat_WalCounters wal_counters,
@@ -1684,6 +1684,54 @@ pg_stat_wal_build_tuple(PgStat_WalCounters wal_counters,
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+/*+ * Returns WAL statistics for a backend with given PID.+ */+Datum+pg_stat_get_backend_wal(PG_FUNCTION_ARGS)+{+ int pid;+ PGPROC *proc;+ ProcNumber procNumber;+ PgStat_Backend *backend_stats;+ PgStat_WalCounters bktype_stats;+ PgBackendStatus *beentry;++ pid = PG_GETARG_INT32(0);+ proc = BackendPidGetProc(pid);++ /*+ * This could be an auxiliary process but these do not report backend+ * statistics due to pgstat_tracks_backend_bktype(), so there is no need+ * for an extra call to AuxiliaryPidGetProc().+ */+ if (!proc)+ PG_RETURN_NULL();++ procNumber = GetNumberFromPGProc(proc);++ beentry = pgstat_get_beentry_by_proc_number(procNumber);+ if (!beentry)+ PG_RETURN_NULL();++ backend_stats = pgstat_fetch_stat_backend(procNumber);+ if (!backend_stats)+ PG_RETURN_NULL();++ /* if PID does not match, leave */+ if (beentry->st_procpid != pid)+ PG_RETURN_NULL();++ /* backend may be gone, so recheck in case */+ if (beentry->st_backendType == B_INVALID)+ PG_RETURN_NULL();++ bktype_stats = backend_stats->wal_counters;++ /* save tuples with data from this PgStat_WalCounters */+ return (pg_stat_wal_build_tuple(bktype_stats, backend_stats->stat_reset_timestamp));+}+
/*
* Returns statistics of WAL activity
*/
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 1e1626964e3..97108a1f4c4 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -5954,6 +5954,13 @@
proargmodes => '{o,o,o,o,o}',
proargnames => '{wal_records,wal_fpi,wal_bytes,wal_buffers_full,stats_reset}',
prosrc => 'pg_stat_get_wal' },
+{ oid => '8037', descr => 'statistics: backend WAL activity',+ proname => 'pg_stat_get_backend_wal', provolatile => 'v',+ proparallel => 'r', prorettype => 'record', proargtypes => 'int4',+ proallargtypes => '{int4,int8,int8,numeric,int8,timestamptz}',+ proargmodes => '{i,o,o,o,o,o}',+ proargnames => '{backend_pid,wal_records,wal_fpi,wal_bytes,wal_buffers_full,stats_reset}',+ prosrc => 'pg_stat_get_backend_wal' },
{ oid => '6248', descr => 'statistics: information about WAL prefetching',
proname => 'pg_stat_get_recovery_prefetch', prorows => '1', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
diff --git a/src/include/pgstat.h b/src/include/pgstat.hindex 459a7cb328e..6b282be4d5b 100644--- a/src/include/pgstat.h+++ b/src/include/pgstat.h@@ -340,12 +340,6 @@ typedef struct PgStat_IO
PgStat_BktypeIO stats[BACKEND_NUM_TYPES];
} PgStat_IO;
-typedef struct PgStat_Backend-{- TimestampTz stat_reset_timestamp;- PgStat_BktypeIO io_stats;-} PgStat_Backend;-
/* ---------
* PgStat_BackendPending Non-flushed backend stats.
* ---------
@@ -488,6 +482,13 @@ typedef struct PgStat_WalStats
TimestampTz stat_reset_timestamp;
} PgStat_WalStats;
+typedef struct PgStat_Backend+{+ TimestampTz stat_reset_timestamp;+ PgStat_BktypeIO io_stats;+ PgStat_WalCounters wal_counters;+} PgStat_Backend;+
/*
* Functions in pgstat.c
*/
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.hindex 06dcea3f0dc..2385839d83f 100644--- a/src/include/utils/pgstat_internal.h+++ b/src/include/utils/pgstat_internal.h@@ -622,7 +622,8 @@ extern void pgstat_archiver_snapshot_cb(void);
/* flags for pgstat_flush_backend() */
#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */
-#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO)+#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */+#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL)
extern bool pgstat_flush_backend(bool nowait, bits32 flags);
extern bool pgstat_backend_flush_cb(bool nowait);
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.outindex 7d91f047bb3..ec056cd00cf 100644--- a/src/test/regress/expected/stats.out+++ b/src/test/regress/expected/stats.out@@ -832,6 +832,8 @@ SELECT sessions > :db_stat_sessions FROM pg_stat_database WHERE datname = (SELEC
SELECT num_requested AS rqst_ckpts_before FROM pg_stat_checkpointer \gset
-- Test pg_stat_wal (and make a temp table so our temp schema exists)
SELECT wal_bytes AS wal_bytes_before FROM pg_stat_wal \gset
+-- Test pg_stat_get_backend_wal (and make a temp table so our temp schema exists)+SELECT wal_bytes AS backend_wal_bytes_before from pg_stat_get_backend_wal(pg_backend_pid()) \gset
CREATE TEMP TABLE test_stats_temp AS SELECT 17;
DROP TABLE test_stats_temp;
-- Checkpoint twice: The checkpointer reports stats after reporting completion
@@ -851,6 +853,18 @@ SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal;
t
(1 row)
+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_backend_pid());+ ?column? +----------+ t+(1 row)+
-- Test pg_stat_get_backend_idset() and some allied functions.
-- In particular, verify that their notion of backend ID matches
-- our temp schema index.
diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sqlindex 11628ebc8a1..92e2ec46a6f 100644--- a/src/test/regress/sql/stats.sql+++ b/src/test/regress/sql/stats.sql@@ -423,6 +423,9 @@ SELECT num_requested AS rqst_ckpts_before FROM pg_stat_checkpointer \gset
-- Test pg_stat_wal (and make a temp table so our temp schema exists)
SELECT wal_bytes AS wal_bytes_before FROM pg_stat_wal \gset
+-- Test pg_stat_get_backend_wal (and make a temp table so our temp schema exists)+SELECT wal_bytes AS backend_wal_bytes_before from pg_stat_get_backend_wal(pg_backend_pid()) \gset+
CREATE TEMP TABLE test_stats_temp AS SELECT 17;
DROP TABLE test_stats_temp;
@@ -435,6 +438,9 @@ CHECKPOINT;
SELECT num_requested > :rqst_ckpts_before FROM pg_stat_checkpointer;
SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal;
+SELECT pg_stat_force_next_flush();+SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_backend_pid());+
-- Test pg_stat_get_backend_idset() and some allied functions.
-- In particular, verify that their notion of backend ID matches
-- our temp schema index.
--
2.34.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bertranddrouvot.pg@gmail.com, michael@paquier.xyz, andres@anarazel.de, byavuz81@gmail.com
Subject: Re: per backend WAL statistics
In-Reply-To: <Z7NSc5j6M4g2r1HD@ip-10-97-1-34.eu-west-3.compute.internal>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox