From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Matthias van de Meent <boekewurm+postgres@gmail.com>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Josef Šimánek <josef.simanek@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: Improvements and additions to COPY progress reporting
Date: Mon, 15 Feb 2021 17:07:11 +0100
Message-ID: <de9df4db-a8e8-ec5f-9540-c5834ab76345@enterprisedb.com> (raw)
In-Reply-To: <CAEze2WhbvkoSm6zs7Jaupe+Ln0ttP6m1Fb3WLodo8-SdTxq6WQ@mail.gmail.com>
References: <CAEze2WiOcgdH4aQA8NtZq-4dgvnJzp8PohdeKchPkhMY-jWZXA@mail.gmail.com>
<CALj2ACW-nY0TAQRzUyKmomwD0udobb53iopFDTn2NytwcSR07w@mail.gmail.com>
<CAEze2Wh753JM+W25grkM30GUXvE_bjHVEJdkdvhnL8SthNSJVA@mail.gmail.com>
<CALj2ACUCASUO9Le2Lq6d_gEdHtVEH6EFG19R-kKTWtRf2TmnVw@mail.gmail.com>
<CAEze2Wi2BS12wj3ZggQwcacW8e9inruaSZpfbWZ40O3Xp_Q-aA@mail.gmail.com>
<CAFp7QwqKwZikq6fdOC7+EShw+ozWzNo63e4PFatcFvAGH4qaag@mail.gmail.com>
<CALj2ACWTEPRJY72NWuwj9eL3=pfx=KVXq4srpe-EwNPO7GKvBA@mail.gmail.com>
<CAEze2WiTSjepqc-JO29yaydeXdgXeqxJ1Hi-DAMvfzKqnnje2w@mail.gmail.com>
<CAEze2Wh_ohsJuRQ7kkCnfZWNqHXLsJCFsKzRauoGNeNgG=eosg@mail.gmail.com>
<CALj2ACWNo1jSQDnGTx+v9jR27BOen59sWdp+j+JrWwuFuhB8Rg@mail.gmail.com>
<CAEze2WhbvkoSm6zs7Jaupe+Ln0ttP6m1Fb3WLodo8-SdTxq6WQ@mail.gmail.com>
Hi,
I agree with these changes in general - I have a couple minor comment:
1) 0001
- the SGML docs are missing a couple tags
- The blocks in copyfrom.cc/copyto.c should be reworked - I don't think
we do this in our codebase. Move the variable declarations to the
beginning, get rid of the out block. Or something like that.
- I fir the "io_target" name misleading, because in some cases it's
actually the *source*.
2) 0002
- I believe "each backend ... reports its" (not theirs), right?
- This seems more like a generic docs improvement, not quite specific to
the COPY progress patch. It's a bit buried, maybe it should be posted
separately. OTOH it's pretty small.
3) 0003
- Some whitespace noise, triggering "git am" warnings.
- Might be good to briefly explain what the regression test does with
the triggers, etc.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:download all (6 files, .tar.gz)v7-0001-Add-progress-reported-components-for-COPY-progress-r.patch (text/x-patch, 10.4K) download
From 32e8bf3e84d58b43c8ab5888eb909e86bdf0ceea Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <boekewurm@gmail.com>
Date: Fri, 12 Feb 2021 14:06:44 +0100
Subject: [PATCH 1/6] Add progress-reported components for COPY progress
reporting
The command, io target and excluded tuple count (by COPY ... FROM ... s'
WHERE -clause) are now reported in the pg_stat_progress_copy view.
Additionally, the column lines_processed is renamed to tuples_processed to
disambiguate the meaning of this column in cases of CSV and BINARY copies and
to stay consistent with regards to names in the pg_stat_progress_*-views.
Of special interest is the reporting of io_target, with which we can
distinguish logical replications' initial table synchronization workers'
progress without having to join the pg_stat_activity view.
---
doc/src/sgml/monitoring.sgml | 37 ++++++++++++++++++++++++++--
src/backend/catalog/system_views.sql | 11 ++++++++-
src/backend/commands/copyfrom.c | 30 +++++++++++++++++++++-
src/backend/commands/copyto.c | 29 ++++++++++++++++++++--
src/include/commands/progress.h | 15 ++++++++++-
src/test/regress/expected/rules.out | 15 ++++++++++-
6 files changed, 129 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgmlindex c602ee4427..3c39c82f1a 100644--- a/doc/src/sgml/monitoring.sgml+++ b/doc/src/sgml/monitoring.sgml@@ -6544,6 +6544,29 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</para></entry>
</row>
+ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>command</structfield> <type>text</type>+ </para>+ <para>+ The command that is running: <literal>COPY FROM</literal>, or+ <literal>COPY TO</literal>.+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>io_target</structfield> <type>text</type>+ </para>+ <para>+ The io target that the data is read from or written to: + <literal>FILE</literal>, <literal>PROGRAM</literal>, + <literal>STDIO</literal> (for COPY FROM STDIN and COPY TO STDOUT),+ or <literal>CALLBACK</literal> (used in the table synchronization+ background worker).+ </para></entry>+ </row>+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>bytes_processed</structfield> <type>bigint</type>
@@ -6565,10 +6588,20 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>lines_processed</structfield> <type>bigint</type>+ <structfield>tuples_processed</structfield> <type>bigint</type>+ </para>+ <para>+ Number of tuples already processed by <command>COPY</command> command.+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>tuples_excluded</structfield> <type>bigint</type>
</para>
<para>
- Number of lines already processed by <command>COPY</command> command.+ Number of tuples not processed because they were excluded by the+ <command>WHERE</command> clause of the <command>COPY</command> command.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sqlindex fa58afd9d7..6a3ac47b85 100644--- a/src/backend/catalog/system_views.sql+++ b/src/backend/catalog/system_views.sql@@ -1129,9 +1129,18 @@ CREATE VIEW pg_stat_progress_copy AS
SELECT
S.pid AS pid, S.datid AS datid, D.datname AS datname,
S.relid AS relid,
+ CASE S.param5 WHEN 1 THEN 'COPY FROM'+ WHEN 2 THEN 'COPY TO'+ END AS command,+ CASE S.param6 WHEN 1 THEN 'FILE'+ WHEN 2 THEN 'PROGRAM'+ WHEN 3 THEN 'STDIO'+ WHEN 4 THEN 'CALLBACK'+ END AS io_target,
S.param1 AS bytes_processed,
S.param2 AS bytes_total,
- S.param3 AS lines_processed+ S.param3 AS tuples_processed,+ S.param4 AS tuples_excluded
FROM pg_stat_get_progress_info('COPY') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.cindex 796ca7b3f7..c3610eb67e 100644--- a/src/backend/commands/copyfrom.c+++ b/src/backend/commands/copyfrom.c@@ -540,6 +540,7 @@ CopyFrom(CopyFromState cstate)
CopyInsertMethod insertMethod;
CopyMultiInsertInfo multiInsertInfo = {0}; /* pacify compiler */
uint64 processed = 0;
+ uint64 excluded = 0;
bool has_before_insert_row_trig;
bool has_instead_insert_row_trig;
bool leafpart_use_multi_insert = false;
@@ -869,7 +870,11 @@ CopyFrom(CopyFromState cstate)
econtext->ecxt_scantuple = myslot;
/* Skip items that don't match COPY's WHERE clause */
if (!ExecQual(cstate->qualexpr, econtext))
+ {+ /* Report that this tuple was filtered out by the WHERE clause */+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_EXCLUDED, ++excluded);
continue;
+ }
}
/* Determine the partition to insert the tuple into */
@@ -1107,7 +1112,7 @@ CopyFrom(CopyFromState cstate)
* for counting tuples inserted by an INSERT command. Update
* progress of the COPY command as well.
*/
- pgstat_progress_update_param(PROGRESS_COPY_LINES_PROCESSED, ++processed);+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED, ++processed);
}
}
@@ -1424,6 +1429,8 @@ BeginCopyFrom(ParseState *pstate,
/* initialize progress */
pgstat_progress_start_command(PROGRESS_COMMAND_COPY,
cstate->rel ? RelationGetRelid(cstate->rel) : InvalidOid);
+ pgstat_progress_update_param(PROGRESS_COPY_COMMAND, PROGRESS_COPY_COMMAND_FROM);+
cstate->bytes_processed = 0;
/* We keep those variables in cstate. */
@@ -1501,6 +1508,27 @@ BeginCopyFrom(ParseState *pstate,
ReceiveCopyBinaryHeader(cstate);
}
+ {+ int64 io_target;+ switch (cstate->copy_src)+ {+ case COPY_FILE:+ if (is_program)+ io_target = PROGRESS_COPY_IO_TARGET_PROGRAM;+ else+ io_target = PROGRESS_COPY_IO_TARGET_FILE;+ break;+ case COPY_OLD_FE:+ case COPY_NEW_FE:+ io_target = PROGRESS_COPY_IO_TARGET_STDIO;+ break;+ case COPY_CALLBACK:+ io_target = PROGRESS_COPY_IO_TARGET_CALLBACK;+ break;+ }+ pgstat_progress_update_param(PROGRESS_COPY_IO_TARGET, io_target);+ }+
/* create workspace for CopyReadAttributes results */
if (!cstate->opts.binary)
{
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.cindex e04ec1e331..42c4a828df 100644--- a/src/backend/commands/copyto.c+++ b/src/backend/commands/copyto.c@@ -772,6 +772,31 @@ BeginCopyTo(ParseState *pstate,
/* initialize progress */
pgstat_progress_start_command(PROGRESS_COMMAND_COPY,
cstate->rel ? RelationGetRelid(cstate->rel) : InvalidOid);
+ {+ const int progress_index[] = {+ PROGRESS_COPY_COMMAND,+ PROGRESS_COPY_IO_TARGET+ };+ int64 progress_vals[] = {+ PROGRESS_COPY_COMMAND_TO,+ 0+ };+ switch (cstate->copy_dest)+ {+ case COPY_FILE:+ if (is_program)+ progress_vals[1] = PROGRESS_COPY_IO_TARGET_PROGRAM;+ else+ progress_vals[1] = PROGRESS_COPY_IO_TARGET_FILE;+ break;+ case COPY_OLD_FE:+ case COPY_NEW_FE:+ progress_vals[1] = PROGRESS_COPY_IO_TARGET_STDIO;+ break;+ }+ pgstat_progress_update_multi_param(2, progress_index, progress_vals);+ }+
cstate->bytes_processed = 0;
MemoryContextSwitchTo(oldcontext);
@@ -954,7 +979,7 @@ CopyTo(CopyToState cstate)
CopyOneRowTo(cstate, slot);
/* Increment amount of processed tuples and update the progress */
- pgstat_progress_update_param(PROGRESS_COPY_LINES_PROCESSED, ++processed);+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED, ++processed);
}
ExecDropSingleTupleTableSlot(slot);
@@ -1321,7 +1346,7 @@ copy_dest_receive(TupleTableSlot *slot, DestReceiver *self)
CopyOneRowTo(cstate, slot);
/* Increment amount of processed tuples and update the progress */
- pgstat_progress_update_param(PROGRESS_COPY_LINES_PROCESSED, ++myState->processed);+ pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED, ++myState->processed);
return true;
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.hindex 95ec5d02e9..e003217554 100644--- a/src/include/commands/progress.h+++ b/src/include/commands/progress.h@@ -136,6 +136,19 @@
/* Commands of PROGRESS_COPY */
#define PROGRESS_COPY_BYTES_PROCESSED 0
#define PROGRESS_COPY_BYTES_TOTAL 1
-#define PROGRESS_COPY_LINES_PROCESSED 2+#define PROGRESS_COPY_TUPLES_PROCESSED 2+#define PROGRESS_COPY_TUPLES_EXCLUDED 3+#define PROGRESS_COPY_COMMAND 4+#define PROGRESS_COPY_IO_TARGET 5++/* Commands of PROGRESS_COPY_COMMAND */+#define PROGRESS_COPY_COMMAND_FROM 1+#define PROGRESS_COPY_COMMAND_TO 2++/* Types of PROGRESS_COPY_INOUT_TYPE */+#define PROGRESS_COPY_IO_TARGET_FILE 1+#define PROGRESS_COPY_IO_TARGET_PROGRAM 2+#define PROGRESS_COPY_IO_TARGET_STDIO 3+#define PROGRESS_COPY_IO_TARGET_CALLBACK 4
#endif
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.outindex 10a1f34ebc..3c6776429d 100644--- a/src/test/regress/expected/rules.out+++ b/src/test/regress/expected/rules.out@@ -1949,9 +1949,22 @@ pg_stat_progress_copy| SELECT s.pid,
s.datid,
d.datname,
s.relid,
+ CASE s.param5+ WHEN 1 THEN 'COPY FROM'::text+ WHEN 2 THEN 'COPY TO'::text+ ELSE NULL::text+ END AS command,+ CASE s.param6+ WHEN 1 THEN 'FILE'::text+ WHEN 2 THEN 'PROGRAM'::text+ WHEN 3 THEN 'STDIO'::text+ WHEN 4 THEN 'CALLBACK'::text+ ELSE NULL::text+ END AS io_target,
s.param1 AS bytes_processed,
s.param2 AS bytes_total,
- s.param3 AS lines_processed+ s.param3 AS tuples_processed,+ s.param4 AS tuples_excluded
FROM (pg_stat_get_progress_info('COPY'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_create_index| SELECT s.pid,
--
2.26.2
v7-0002-0001-review.patch (text/x-patch, 1.9K) download
From ddcc5f263958efefda871e03f17715aa6111a2dc Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Mon, 15 Feb 2021 16:26:32 +0100
Subject: [PATCH 2/6] 0001 review
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/commands/copyfrom.c | 1 +
src/backend/commands/copyto.c | 2 ++
3 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgmlindex 3c39c82f1a..c5048fefd4 100644--- a/doc/src/sgml/monitoring.sgml+++ b/doc/src/sgml/monitoring.sgml@@ -6561,7 +6561,7 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<para>
The io target that the data is read from or written to:
<literal>FILE</literal>, <literal>PROGRAM</literal>,
- <literal>STDIO</literal> (for COPY FROM STDIN and COPY TO STDOUT),+ <literal>STDIO</literal> (for <command>COPY FROM STDIN</command> and <command>COPY TO STDOUT</command>),
or <literal>CALLBACK</literal> (used in the table synchronization
background worker).
</para></entry>
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.cindex c3610eb67e..d98f447cf5 100644--- a/src/backend/commands/copyfrom.c+++ b/src/backend/commands/copyfrom.c@@ -1508,6 +1508,7 @@ BeginCopyFrom(ParseState *pstate,
ReceiveCopyBinaryHeader(cstate);
}
+ /* XXX this block seems weird */
{
int64 io_target;
switch (cstate->copy_src)
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.cindex 42c4a828df..4b76bf5ef6 100644--- a/src/backend/commands/copyto.c+++ b/src/backend/commands/copyto.c@@ -772,6 +772,8 @@ BeginCopyTo(ParseState *pstate,
/* initialize progress */
pgstat_progress_start_command(PROGRESS_COMMAND_COPY,
cstate->rel ? RelationGetRelid(cstate->rel) : InvalidOid);
++ /* XXX this too */
{
const int progress_index[] = {
PROGRESS_COPY_COMMAND,
--
2.26.2
v7-0003-Add-backlinks-to-progress-reporting-documentation.patch (text/x-patch, 7.4K) download
From c2a92e0fee760e52a604f6eb11e0766f5ffdf090 Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <boekewurm@gmail.com>
Date: Fri, 5 Feb 2021 23:11:50 +0100
Subject: [PATCH 3/6] Add backlinks to progress reporting documentation
Previously, for most progress-reported features, the only place the
feature was mentioned is in the progress reporting document itself.
This makes the progress reporting more discoverable from the reported
commands.
---
doc/src/sgml/ref/analyze.sgml | 7 +++++++
doc/src/sgml/ref/cluster.sgml | 6 ++++++
doc/src/sgml/ref/copy.sgml | 14 ++++++++++++++
doc/src/sgml/ref/create_index.sgml | 8 ++++++++
doc/src/sgml/ref/pg_basebackup.sgml | 1 +
doc/src/sgml/ref/reindex.sgml | 7 +++++++
doc/src/sgml/ref/vacuum.sgml | 11 +++++++++++
7 files changed, 54 insertions(+)
diff --git a/doc/src/sgml/ref/analyze.sgml b/doc/src/sgml/ref/analyze.sgmlindex 7d816c87c6..9db9070b62 100644--- a/doc/src/sgml/ref/analyze.sgml+++ b/doc/src/sgml/ref/analyze.sgml@@ -273,6 +273,12 @@ ANALYZE [ VERBOSE ] [ <replaceable class="parameter">table_and_columns</replacea
will not record new statistics for that table. Any existing statistics
will be retained.
</para>
++ <para>+ Each backend running the <command>ANALYZE</command> command will report their+ progress to the <structname>pg_stat_progress_analyze</structname> view.+ See <xref linkend="analyze-progress-reporting"/> for details.+ </para>
</refsect1>
<refsect1>
@@ -291,6 +297,7 @@ ANALYZE [ VERBOSE ] [ <replaceable class="parameter">table_and_columns</replacea
<member><xref linkend="app-vacuumdb"/></member>
<member><xref linkend="runtime-config-resource-vacuum-cost"/></member>
<member><xref linkend="autovacuum"/></member>
+ <member><xref linkend="analyze-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/cluster.sgml b/doc/src/sgml/ref/cluster.sgmlindex 5dd21a0189..5c2270f71b 100644--- a/doc/src/sgml/ref/cluster.sgml+++ b/doc/src/sgml/ref/cluster.sgml@@ -192,6 +192,11 @@ CLUSTER [VERBOSE]
are periodically reclustered.
</para>
+ <para>+ Each backend running the <command>CLUSTER</command> command will report their+ progress to the <structname>pg_stat_progress_cluster</structname> view.+ See <xref linkend="cluster-progress-reporting"/> for details.+ </para>
</refsect1>
<refsect1>
@@ -242,6 +247,7 @@ CLUSTER <replaceable class="parameter">index_name</replaceable> ON <replaceable
<simplelist type="inline">
<member><xref linkend="app-clusterdb"/></member>
+ <member><xref linkend="cluster-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgmlindex 0fca6583af..af3ce72561 100644--- a/doc/src/sgml/ref/copy.sgml+++ b/doc/src/sgml/ref/copy.sgml@@ -82,6 +82,12 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
specified, data is transmitted via the connection between the
client and the server.
</para>
++ <para>+ Each backend running the <command>COPY</command> command will report their+ progress to the <structname>pg_stat_progress_copy</structname> view.+ See <xref linkend="copy-progress-reporting"/> for details.+ </para>
</refsect1>
<refsect1>
@@ -1052,4 +1058,12 @@ COPY [ BINARY ] <replaceable class="parameter">table_name</replaceable>
[ WITH NULL AS '<replaceable class="parameter">null_string</replaceable>' ]
</synopsis></para>
</refsect1>
+ + <refsect1>+ <title>See Also</title>+ + <simplelist type="inline">+ <member><xref linkend="copy-progress-reporting"/></member>+ </simplelist>+ </refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/src/sgml/ref/create_index.sgmlindex a5271a9f8f..278058f500 100644--- a/doc/src/sgml/ref/create_index.sgml+++ b/doc/src/sgml/ref/create_index.sgml@@ -865,6 +865,13 @@ Indexes:
will interpret it as <literal>USING gist</literal>, to simplify conversion
of old databases to GiST.
</para>
++ <para>+ Each backend running the <command>CREATE INDEX</command> command will + report their progress to the + <structname>pg_stat_progress_create_index</structname> view.+ See <xref linkend="create-index-progress-reporting"/> for details.+ </para>
</refsect1>
<refsect1>
@@ -978,6 +985,7 @@ CREATE INDEX CONCURRENTLY sales_quantity_index ON sales_table (quantity);
<member><xref linkend="sql-alterindex"/></member>
<member><xref linkend="sql-dropindex"/></member>
<member><xref linkend="sql-reindex"/></member>
+ <member><xref linkend="create-index-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgmlindex 5754ad5aa6..14cc88a852 100644--- a/doc/src/sgml/ref/pg_basebackup.sgml+++ b/doc/src/sgml/ref/pg_basebackup.sgml@@ -904,6 +904,7 @@ PostgreSQL documentation
<simplelist type="inline">
<member><xref linkend="app-pgdump"/></member>
+ <member><xref linkend="basebackup-progress-reporting"/></member>
</simplelist>
</refsect1>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgmlindex 07795b5737..f69f5db403 100644--- a/doc/src/sgml/ref/reindex.sgml+++ b/doc/src/sgml/ref/reindex.sgml@@ -493,6 +493,12 @@ Indexes:
is reindexed concurrently, those indexes will be skipped. (It is possible
to reindex such indexes without the <command>CONCURRENTLY</command> option.)
</para>
++ <para>+ Each backend running the <command>REINDEX</command> command will report+ their progress to the <structname>pg_stat_progress_create_index</structname>+ view. See <xref linkend="create-index-progress-reporting"/> for details.+ </para>
</refsect2>
</refsect1>
@@ -551,6 +557,7 @@ REINDEX TABLE CONCURRENTLY my_broken_table;
<member><xref linkend="sql-createindex"/></member>
<member><xref linkend="sql-dropindex"/></member>
<member><xref linkend="app-reindexdb"/></member>
+ <member><xref linkend="create-index-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgmlindex 4bb624979b..bab3d00313 100644--- a/doc/src/sgml/ref/vacuum.sgml+++ b/doc/src/sgml/ref/vacuum.sgml@@ -393,6 +393,15 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
information about automatic and manual vacuuming, see
<xref linkend="routine-vacuuming"/>.
</para>
+ <para>+ Each backend running the <command>VACUUM</command> command without the+ <literal>FULL</literal> option will report their progress in the + <structname>pg_stat_progress_vacuum</structname> view. Backends running+ <command>VACUUM</command> with the <literal>FULL</literal> option report + progress in the <structname>pg_stat_progress_cluster</structname> instead.+ See <xref linkend="vacuum-progress-reporting"/> and + <xref linkend="cluster-progress-reporting"/> for details.+ </para>
</refsect1>
<refsect1>
@@ -422,6 +431,8 @@ VACUUM (VERBOSE, ANALYZE) onek;
<member><xref linkend="app-vacuumdb"/></member>
<member><xref linkend="runtime-config-resource-vacuum-cost"/></member>
<member><xref linkend="autovacuum"/></member>
+ <member><xref linkend="vacuum-progress-reporting"/></member>+ <member><xref linkend="cluster-progress-reporting"/></member>
</simplelist>
</refsect1>
</refentry>
--
2.26.2
v7-0004-0002-review.patch (text/x-patch, 4.1K) download
From d0d3e87d5c52d894c93c434ff7d0e03055948a1c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Mon, 15 Feb 2021 16:31:58 +0100
Subject: [PATCH 4/6] 0002 review
---
doc/src/sgml/ref/analyze.sgml | 2 +-
doc/src/sgml/ref/cluster.sgml | 2 +-
doc/src/sgml/ref/copy.sgml | 2 +-
doc/src/sgml/ref/create_index.sgml | 2 +-
doc/src/sgml/ref/reindex.sgml | 2 +-
doc/src/sgml/ref/vacuum.sgml | 2 +-
6 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/ref/analyze.sgml b/doc/src/sgml/ref/analyze.sgmlindex 9db9070b62..5a9e3322b3 100644--- a/doc/src/sgml/ref/analyze.sgml+++ b/doc/src/sgml/ref/analyze.sgml@@ -275,7 +275,7 @@ ANALYZE [ VERBOSE ] [ <replaceable class="parameter">table_and_columns</replacea
</para>
<para>
- Each backend running the <command>ANALYZE</command> command will report their+ Each backend running the <command>ANALYZE</command> command will report its
progress to the <structname>pg_stat_progress_analyze</structname> view.
See <xref linkend="analyze-progress-reporting"/> for details.
</para>
diff --git a/doc/src/sgml/ref/cluster.sgml b/doc/src/sgml/ref/cluster.sgmlindex 5c2270f71b..70c4b33454 100644--- a/doc/src/sgml/ref/cluster.sgml+++ b/doc/src/sgml/ref/cluster.sgml@@ -193,7 +193,7 @@ CLUSTER [VERBOSE]
</para>
<para>
- Each backend running the <command>CLUSTER</command> command will report their+ Each backend running the <command>CLUSTER</command> command will report its
progress to the <structname>pg_stat_progress_cluster</structname> view.
See <xref linkend="cluster-progress-reporting"/> for details.
</para>
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgmlindex af3ce72561..41e1d7c371 100644--- a/doc/src/sgml/ref/copy.sgml+++ b/doc/src/sgml/ref/copy.sgml@@ -84,7 +84,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</para>
<para>
- Each backend running the <command>COPY</command> command will report their+ Each backend running the <command>COPY</command> command will report its
progress to the <structname>pg_stat_progress_copy</structname> view.
See <xref linkend="copy-progress-reporting"/> for details.
</para>
diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/src/sgml/ref/create_index.sgmlindex 278058f500..6079a0ef28 100644--- a/doc/src/sgml/ref/create_index.sgml+++ b/doc/src/sgml/ref/create_index.sgml@@ -868,7 +868,7 @@ Indexes:
<para>
Each backend running the <command>CREATE INDEX</command> command will
- report their progress to the + report it's progress to the
<structname>pg_stat_progress_create_index</structname> view.
See <xref linkend="create-index-progress-reporting"/> for details.
</para>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgmlindex f69f5db403..3b4d409856 100644--- a/doc/src/sgml/ref/reindex.sgml+++ b/doc/src/sgml/ref/reindex.sgml@@ -496,7 +496,7 @@ Indexes:
<para>
Each backend running the <command>REINDEX</command> command will report
- their progress to the <structname>pg_stat_progress_create_index</structname>+ its progress to the <structname>pg_stat_progress_create_index</structname>
view. See <xref linkend="create-index-progress-reporting"/> for details.
</para>
</refsect2>
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgmlindex bab3d00313..0624235696 100644--- a/doc/src/sgml/ref/vacuum.sgml+++ b/doc/src/sgml/ref/vacuum.sgml@@ -395,7 +395,7 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
</para>
<para>
Each backend running the <command>VACUUM</command> command without the
- <literal>FULL</literal> option will report their progress in the + <literal>FULL</literal> option will report its progress in the
<structname>pg_stat_progress_vacuum</structname> view. Backends running
<command>VACUUM</command> with the <literal>FULL</literal> option report
progress in the <structname>pg_stat_progress_cluster</structname> instead.
--
2.26.2
v7-0005-Add-copy-progress-reporting-regression-tests.patch (text/x-patch, 4.7K) download
From c76d1d779b797a42ebc58abe7ead4f5d80b45bcb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Josef=20=C5=A0im=C3=A1nek?= <josef.simanek@gmail.com>
Date: Tue, 9 Feb 2021 13:06:45 +0100
Subject: [PATCH 5/6] Add copy progress reporting regression tests.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This tests some basic features of copy progress reporting.
Sadly, we can only request one snapshot of progress_reporting each
transaction / statement, so we can't (easily) get intermediate results without
each result requiring a seperate statement being run.
Author: Josef Šimánek, Matthias van de Meent
---
src/test/regress/input/copy.source | 57 +++++++++++++++++++++++++++++
src/test/regress/output/copy.source | 44 ++++++++++++++++++++++
2 files changed, 101 insertions(+)
diff --git a/src/test/regress/input/copy.source b/src/test/regress/input/copy.sourceindex a1d529ad36..ddde33e7cc 100644--- a/src/test/regress/input/copy.source+++ b/src/test/regress/input/copy.source@@ -201,3 +201,60 @@ select * from parted_copytest where b = 1;
select * from parted_copytest where b = 2;
drop table parted_copytest;
++--+-- progress reporting+-- ++-- setup+-- reuse employer datatype, that has a small sized data set++create table progress_reporting (+ name text,+ age int4,+ location point,+ salary int4,+ manager name+);++create function notice_after_progress_reporting() returns trigger AS+$$+declare report record;+begin+ -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to + -- variance in system process ids, and concurrency in runs of tests.+ -- Additionally, due to the usage of this test in pg_regress, the 'datid'+ -- also is not consistent between runs.+ select into report (to_jsonb(r) - '{pid,relid,datid}'::text[]) as value+ from pg_stat_progress_copy r+ where pid = pg_backend_pid();++ raise info 'progress: %', report.value::text;++ RETURN NEW;+END;+$$ LANGUAGE plpgsql;++create trigger check_after_progress_reporting+ after insert on progress_reporting+ for each statement+ execute function notice_after_progress_reporting();++-- reporting of STDIO imports, and correct bytes-processed/tuples-processed reporting++copy progress_reporting from stdin;+sharon 25 (15,12) 1000 sam+sam 30 (10,5) 2000 bill+bill 20 (11,10) 1000 sharon+\.++-- reporting of FILE imports, and correct reporting of tuples-excluded++copy progress_reporting from '@abs_srcdir@/data/emp.data'+ where (salary < 2000);++-- cleanup progress_reporting++drop trigger check_after_progress_reporting on progress_reporting;+drop function notice_after_progress_reporting();+drop table progress_reporting;diff --git a/src/test/regress/output/copy.source b/src/test/regress/output/copy.sourceindex 938d3551da..60f4206aa1 100644--- a/src/test/regress/output/copy.source+++ b/src/test/regress/output/copy.source@@ -165,3 +165,47 @@ select * from parted_copytest where b = 2;
(1 row)
drop table parted_copytest;
+--+-- progress reporting+-- +-- setup+-- reuse employer datatype, that has a small sized data set+create table progress_reporting (+ name text,+ age int4,+ location point,+ salary int4,+ manager name+);+create function notice_after_progress_reporting() returns trigger AS+$$+declare report record;+begin+ -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to + -- variance in system process ids, and concurrency in runs of tests.+ -- Additionally, due to the usage of this test in pg_regress, the 'datid'+ -- also is not consistent between runs.+ select into report (to_jsonb(r) - '{pid,relid,datid}'::text[]) as value+ from pg_stat_progress_copy r+ where pid = pg_backend_pid();++ raise info 'progress: %', report.value::text;++ RETURN NEW;+END;+$$ LANGUAGE plpgsql;+create trigger check_after_progress_reporting+ after insert on progress_reporting+ for each statement+ execute function notice_after_progress_reporting();+-- reporting of STDIO imports, and correct bytes-processed/tuples-processed reporting+copy progress_reporting from stdin;+INFO: progress: {"command": "COPY FROM", "datname": "regression", "io_target": "STDIO", "bytes_total": 0, "bytes_processed": 79, "tuples_excluded": 0, "tuples_processed": 3}+-- reporting of FILE imports, and correct reporting of tuples-excluded+copy progress_reporting from '@abs_srcdir@/data/emp.data'+ where (salary < 2000);+INFO: progress: {"command": "COPY FROM", "datname": "regression", "io_target": "FILE", "bytes_total": 79, "bytes_processed": 79, "tuples_excluded": 1, "tuples_processed": 2}+-- cleanup progress_reporting+drop trigger check_after_progress_reporting on progress_reporting;+drop function notice_after_progress_reporting();+drop table progress_reporting;--
2.26.2
v7-0006-0003-review.patch (text/x-patch, 2.0K) download
From 07d065c2a127301011849e6bd35b8062c38e915b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Mon, 15 Feb 2021 16:36:28 +0100
Subject: [PATCH 6/6] 0003 review
---
src/test/regress/input/copy.source | 4 ++--
src/test/regress/output/copy.source | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/test/regress/input/copy.source b/src/test/regress/input/copy.sourceindex ddde33e7cc..4ee1b26326 100644--- a/src/test/regress/input/copy.source+++ b/src/test/regress/input/copy.source@@ -204,7 +204,7 @@ drop table parted_copytest;
--
-- progress reporting
--- +--
-- setup
-- reuse employer datatype, that has a small sized data set
@@ -221,7 +221,7 @@ create function notice_after_progress_reporting() returns trigger AS
$$
declare report record;
begin
- -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to + -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to
-- variance in system process ids, and concurrency in runs of tests.
-- Additionally, due to the usage of this test in pg_regress, the 'datid'
-- also is not consistent between runs.
diff --git a/src/test/regress/output/copy.source b/src/test/regress/output/copy.sourceindex 60f4206aa1..8ebfb7ae81 100644--- a/src/test/regress/output/copy.source+++ b/src/test/regress/output/copy.source@@ -167,7 +167,7 @@ select * from parted_copytest where b = 2;
drop table parted_copytest;
--
-- progress reporting
--- +--
-- setup
-- reuse employer datatype, that has a small sized data set
create table progress_reporting (
@@ -181,7 +181,7 @@ create function notice_after_progress_reporting() returns trigger AS
$$
declare report record;
begin
- -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to + -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to
-- variance in system process ids, and concurrency in runs of tests.
-- Additionally, due to the usage of this test in pg_regress, the 'datid'
-- also is not consistent between runs.
--
2.26.2
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@enterprisedb.com, boekewurm+postgres@gmail.com, bharath.rupireddyforpostgres@gmail.com, josef.simanek@gmail.com
Subject: Re: Improvements and additions to COPY progress reporting
In-Reply-To: <de9df4db-a8e8-ec5f-9540-c5834ab76345@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox