Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h0kt5-0006fq-Fi for pgsql-hackers@arkaria.postgresql.org; Mon, 04 Mar 2019 10:31:59 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1h0kt4-0004xq-4J for pgsql-hackers@arkaria.postgresql.org; Mon, 04 Mar 2019 10:31:58 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h0kt3-0004xj-Rd for pgsql-hackers@lists.postgresql.org; Mon, 04 Mar 2019 10:31:57 +0000 Received: from tama500.ecl.ntt.co.jp ([129.60.39.148]) by magus.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1h0kt0-0001Nq-7g for pgsql-hackers@postgresql.org; Mon, 04 Mar 2019 10:31:57 +0000 Received: from vc2.ecl.ntt.co.jp (vc2.ecl.ntt.co.jp [129.60.86.154]) by tama500.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x24AVi80010288; Mon, 4 Mar 2019 19:31:44 +0900 Received: from vc2.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id 20AB463940D; Mon, 4 Mar 2019 19:31:44 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id 14AE46393AD; Mon, 4 Mar 2019 19:31:44 +0900 (JST) Received: from [127.0.0.1] (unknown [129.60.241.68]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id 0B30E4003FE; Mon, 4 Mar 2019 19:31:44 +0900 (JST) From: Tatsuro Yamada Subject: Re: [HACKERS] CLUSTER command progress monitor References: <20181218172311.qjrcso7cozfi7j2d@alvherre.pgsql> <58bbcaf7-a3e9-3718-51f1-65b4d1af97e2@lab.ntt.co.jp> <76d84ded-c81c-98b1-6fa4-4551ff295008@lab.ntt.co.jp> Message-ID: Date: Mon, 4 Mar 2019 19:31:21 +0900 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/mixed; boundary="------------533B1B7756EA5C06EE5F72FE" Content-Language: en-US X-CC-Mail-RelayStamp: 1 To: Robert Haas Cc: Alvaro Herrera , Dmitry Dolgov <9erthalion6@gmail.com>, Peter Geoghegan , PostgreSQL-development , Jeff Janes , Thomas Munro , Michael Paquier X-TM-AS-MML: disable List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------533B1B7756EA5C06EE5F72FE Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 2019/03/02 4:15, Robert Haas wrote: > On Thu, Feb 28, 2019 at 11:54 PM Tatsuro Yamada > wrote: >> Attached patch is wip patch. Thanks for your comments! :) I revised the code and the document. > + CLUSTER and VACUUM FULL, > showing current progress. > > and -> or Fixed. > + certain commands during command execution. Currently, the suppoted > + progress reporting commands are VACUUM and > CLUSTER. > > suppoted -> supported > > But I'd just say: Currently, the only commands which support progress > reporting are VACUUM and > CLUSTER. I choose the latter. Fixed. > + Running VACUUM FULL is listed in > pg_stat_progress_cluster > + view because it uses CLUSTER command > internally. See . > > How about: Running VACUUM FULL is listed in > pg_stat_progress_cluster because both > VACUUM FULL and CLUSTER rewrite > the table, while regular VACUUM only modifies it in > place. Fixed. > + Current processing command: CLUSTER/VACUUM FULL. > > The command that is running. Either CLUSTER or VACUUM FULL. Fixed. > + Current processing phase of cluster/vacuum full. See linkend='cluster-phases'> or . > > Current processing phase of CLUSTER or VACUUM FULL. > > Or maybe better, just abbreviate to: Current processing phase. Fixed as you suggested. > + Scan method of table: index scan/seq scan. > > Eh, shouldn't this be gone now? And likewise for the view definition? Fixed. Sorry, It was an oversight. > + OID of the index. > > If the table is being scanned using an index, this is the OID of the > index being used; otherwise, it is zero. Fixed. > + heap_tuples_total > > Leftovers. Skipping over the rest of your documentation changes since > it looks like a bunch of things there still need to be updated. I agree. Thanks a lot! I'll divide the patch into two patch such as code and document. > + pgstat_progress_start_command(PROGRESS_COMMAND_CLUSTER, tableOid); > > This now appears inside cluster_rel(), but also vacuum_rel() is still > doing the same thing. That's wrong. It was an oversight too. I fixed. > + if(OidIsValid(indexOid)) > > Missing space. Please pgindent. Fixed. I Will do pgindent later. Please find attached files. :) Thanks, Tatsuro Yamada --------------533B1B7756EA5C06EE5F72FE Content-Type: text/x-patch; name="progress_monitor_for_cluster_command_v7_code.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="progress_monitor_for_cluster_command_v7_code.patch" diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 3e229c693c..0d0f8f0e31 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -906,6 +906,30 @@ CREATE VIEW pg_stat_progress_vacuum AS FROM pg_stat_get_progress_info('VACUUM') AS S LEFT JOIN pg_database D ON S.datid = D.oid; +CREATE VIEW pg_stat_progress_cluster AS + SELECT + S.pid AS pid, + S.datid AS datid, + D.datname AS datname, + S.relid AS relid, + CASE S.param1 WHEN 1 THEN 'CLUSTER' + WHEN 2 THEN 'VACUUM FULL' + END AS command, + CASE S.param2 WHEN 0 THEN 'initializing' + WHEN 1 THEN 'seq scanning heap' + WHEN 2 THEN 'index scanning heap' + WHEN 3 THEN 'sorting tuples' + WHEN 4 THEN 'writing new heap' + WHEN 5 THEN 'swapping relation files' + WHEN 6 THEN 'rebuilding index' + WHEN 7 THEN 'performing final cleanup' + END AS phase, + S.param3 AS cluster_index_relid, + S.param4 AS heap_tuples_scanned, + S.param5 AS heap_tuples_vacuumed + FROM pg_stat_get_progress_info('CLUSTER') AS S + LEFT JOIN pg_database D ON S.datid = D.oid; + CREATE VIEW pg_user_mappings AS SELECT U.oid AS umid, diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index a74af4c171..f22ff590f0 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -35,10 +35,12 @@ #include "catalog/objectaccess.h" #include "catalog/toasting.h" #include "commands/cluster.h" +#include "commands/progress.h" #include "commands/tablecmds.h" #include "commands/vacuum.h" #include "miscadmin.h" #include "optimizer/optimizer.h" +#include "pgstat.h" #include "storage/bufmgr.h" #include "storage/lmgr.h" #include "storage/predicate.h" @@ -275,6 +277,8 @@ cluster_rel(Oid tableOid, Oid indexOid, int options) /* Check for user-requested abort. */ CHECK_FOR_INTERRUPTS(); + pgstat_progress_start_command(PROGRESS_COMMAND_CLUSTER, tableOid); + /* * We grab exclusive access to the target rel and index for the duration * of the transaction. (This is redundant for the single-transaction @@ -385,6 +389,18 @@ cluster_rel(Oid tableOid, Oid indexOid, int options) */ CheckTableNotInUse(OldHeap, OidIsValid(indexOid) ? "CLUSTER" : "VACUUM"); + /* Set command to column */ + if (OidIsValid(indexOid)) + { + pgstat_progress_update_param(PROGRESS_CLUSTER_COMMAND, + PROGRESS_CLUSTER_COMMAND_CLUSTER); + } + else + { + pgstat_progress_update_param(PROGRESS_CLUSTER_COMMAND, + PROGRESS_CLUSTER_COMMAND_VACUUM_FULL); + } + /* Check heap and index are valid to cluster on */ if (OidIsValid(indexOid)) check_index_is_clusterable(OldHeap, indexOid, recheck, AccessExclusiveLock); @@ -415,6 +431,8 @@ cluster_rel(Oid tableOid, Oid indexOid, int options) rebuild_relation(OldHeap, indexOid, verbose); /* NB: rebuild_relation does table_close() on OldHeap */ + + pgstat_progress_end_command(); } /* @@ -923,12 +941,26 @@ copy_heap_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose, */ if (OldIndex != NULL && !use_sort) { + const int ci_index[] = { + PROGRESS_CLUSTER_PHASE, + PROGRESS_CLUSTER_INDEX_RELID + }; + int64 ci_val[2]; + + /* Set phase and OIDOldIndex to columns */ + ci_val[0] = PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP; + ci_val[1] = OIDOldIndex; + pgstat_progress_update_multi_param(2, ci_index, ci_val); + heapScan = NULL; indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, 0, 0); index_rescan(indexScan, NULL, 0, NULL, 0); } else { + pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, + PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP); + heapScan = heap_beginscan(OldHeap, SnapshotAny, 0, (ScanKey) NULL); indexScan = NULL; } @@ -1049,6 +1081,11 @@ copy_heap_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose, tups_vacuumed += 1; tups_recently_dead -= 1; } + + /* set tups_vacuumed column for VACUUM FULL */ + pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_VACUUMED, + tups_vacuumed); + continue; } @@ -1060,6 +1097,10 @@ copy_heap_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose, oldTupDesc, newTupDesc, values, isnull, rwstate); + + /* Regardless of index scan or seq scan, update tuples_scanned column */ + pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED, + num_tuples); } if (indexScan != NULL) @@ -1073,8 +1114,25 @@ copy_heap_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose, */ if (tuplesort != NULL) { + double num_tuples = 0; + const int cp_index[] = { + PROGRESS_CLUSTER_PHASE, + PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED + }; + int64 cp_val[2]; + + /* Report that we are now sorting tuples */ + cp_val[0] = PROGRESS_CLUSTER_PHASE_SORT_TUPLES; + cp_val[1] = num_tuples; + pgstat_progress_update_multi_param(2, cp_index, cp_val); + tuplesort_performsort(tuplesort); + /* Report that we are now writing new heap */ + cp_val[0] = PROGRESS_CLUSTER_PHASE_WRITE_NEW_HEAP; + cp_val[1] = num_tuples; + pgstat_progress_update_multi_param(2, cp_index, cp_val); + for (;;) { HeapTuple tuple; @@ -1085,10 +1143,14 @@ copy_heap_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose, if (tuple == NULL) break; + num_tuples += 1; reform_and_rewrite_tuple(tuple, oldTupDesc, newTupDesc, values, isnull, rwstate); + + pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED, + num_tuples); } tuplesort_end(tuplesort); @@ -1526,6 +1588,16 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap, Oid mapped_tables[4]; int reindex_flags; int i; + const int cp_index[] = { + PROGRESS_CLUSTER_PHASE, + PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED + }; + int64 cp_val[2]; + + /* Report that we are now swapping relation files */ + cp_val[0] = PROGRESS_CLUSTER_PHASE_SWAP_REL_FILES; + cp_val[1] = 0; + pgstat_progress_update_multi_param(2, cp_index, cp_val); /* Zero out possible results from swapped_relation_files */ memset(mapped_tables, 0, sizeof(mapped_tables)); @@ -1561,6 +1633,11 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap, * because the new heap won't contain any HOT chains at all, let alone * broken ones, so it can't be necessary to set indcheckxmin. */ + + /* Report that we are now reindexing relations */ + pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, + PROGRESS_CLUSTER_PHASE_REBUILD_INDEX); + reindex_flags = REINDEX_REL_SUPPRESS_INDEX_USE; if (check_constraints) reindex_flags |= REINDEX_REL_CHECK_CONSTRAINTS; @@ -1576,6 +1653,10 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap, reindex_relation(OIDOldHeap, reindex_flags, 0); + /* Report that we are now doing clean up */ + pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE, + PROGRESS_CLUSTER_PHASE_FINAL_CLEANUP); + /* * If the relation being rebuild is pg_class, swap_relation_files() * couldn't update pg_class's own pg_class entry (check comments in diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 69f7265779..37ff3dbff6 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS) /* Translate command name into command type code. */ if (pg_strcasecmp(cmd, "VACUUM") == 0) cmdtype = PROGRESS_COMMAND_VACUUM; + else if(pg_strcasecmp(cmd, "CLUSTER") == 0) + cmdtype = PROGRESS_COMMAND_CLUSTER; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h index 9858b36a38..480f2e6820 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -34,4 +34,24 @@ #define PROGRESS_VACUUM_PHASE_TRUNCATE 5 #define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6 +/* Progress parameters for cluster */ +#define PROGRESS_CLUSTER_COMMAND 0 +#define PROGRESS_CLUSTER_PHASE 1 +#define PROGRESS_CLUSTER_INDEX_RELID 2 +#define PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED 3 +#define PROGRESS_CLUSTER_HEAP_TUPLES_VACUUMED 4 + +/* Phases of cluster (as dvertised via PROGRESS_CLUSTER_PHASE) */ +#define PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP 1 +#define PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP 2 +#define PROGRESS_CLUSTER_PHASE_SORT_TUPLES 3 +#define PROGRESS_CLUSTER_PHASE_WRITE_NEW_HEAP 4 +#define PROGRESS_CLUSTER_PHASE_SWAP_REL_FILES 5 +#define PROGRESS_CLUSTER_PHASE_REBUILD_INDEX 6 +#define PROGRESS_CLUSTER_PHASE_FINAL_CLEANUP 7 + +/* Commands of PROGRESS_CLUSTER */ +#define PROGRESS_CLUSTER_COMMAND_CLUSTER 1 +#define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL 2 + #endif diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 88a75fb798..745685c8a6 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -934,7 +934,8 @@ typedef enum typedef enum ProgressCommandType { PROGRESS_COMMAND_INVALID, - PROGRESS_COMMAND_VACUUM + PROGRESS_COMMAND_VACUUM, + PROGRESS_COMMAND_CLUSTER } ProgressCommandType; #define PGSTAT_NUM_PROGRESS_PARAM 10 diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 98f417cb57..e102e91172 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1829,6 +1829,31 @@ pg_stat_database_conflicts| SELECT d.oid AS datid, pg_stat_get_db_conflict_bufferpin(d.oid) AS confl_bufferpin, pg_stat_get_db_conflict_startup_deadlock(d.oid) AS confl_deadlock FROM pg_database d; +pg_stat_progress_cluster| SELECT s.pid, + s.datid, + d.datname, + s.relid, + CASE s.param1 + WHEN 1 THEN 'CLUSTER'::text + WHEN 2 THEN 'VACUUM FULL'::text + ELSE NULL::text + END AS command, + CASE s.param2 + WHEN 0 THEN 'initializing'::text + WHEN 1 THEN 'seq scanning heap'::text + WHEN 2 THEN 'index scanning heap'::text + WHEN 3 THEN 'sorting tuples'::text + WHEN 4 THEN 'writing new heap'::text + WHEN 5 THEN 'swapping relation files'::text + WHEN 6 THEN 'rebuilding index'::text + WHEN 7 THEN 'performing final cleanup'::text + ELSE NULL::text + END AS phase, + s.param4 AS cluster_index_relid, + s.param5 AS heap_tuples_scanned, + s.param6 AS heap_tuples_vacuumed, + FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_progress_vacuum| SELECT s.pid, s.datid, d.datname, --------------533B1B7756EA5C06EE5F72FE Content-Type: text/x-patch; name="progress_monitor_for_cluster_command_v7_doc.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="progress_monitor_for_cluster_command_v7_doc.patch" diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 0e73cdcdda..178e21fc1f 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -344,6 +344,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser + + pg_stat_progress_clusterpg_stat_progress_cluster + One row for each backend running + CLUSTER or VACUUM FULL, showing current progress. + See . + + + @@ -3376,9 +3384,9 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, PostgreSQL has the ability to report the progress of - certain commands during command execution. Currently, the only command - which supports progress reporting is VACUUM. This may be - expanded in the future. + certain commands during command execution. Currently, the only commands which + support progress reporting are VACUUM and + CLUSTER. This may be expanded in the future. @@ -3390,9 +3398,10 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, one row for each backend (including autovacuum worker processes) that is currently vacuuming. The tables below describe the information that will be reported and provide information about how to interpret it. - Progress reporting is not currently supported for VACUUM FULL - and backends running VACUUM FULL will not be listed in this - view. + Running VACUUM FULL is listed in pg_stat_progress_cluster + because both VACUUM FULL and CLUSTER + rewrite the table, while regular VACUUM only modifies it + in place. See . @@ -3569,6 +3578,202 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
+
+ + + CLUSTER Progress Reporting + + + Whenever CLUSTER is running, the + pg_stat_progress_cluster view will contain + a row for each backend that is currently running CLUSTER or VACUUM FULL. + The tables below describe the information that will be reported and + provide information about how to interpret it. + + + + <structname>pg_stat_progress_cluster</structname> View + + + + Column + Type + Description + + + + + + pid + integer + Process ID of backend. + + + datid + oid + OID of the database to which this backend is connected. + + + datname + name + Name of the database to which this backend is connected. + + + relid + oid + OID of the table being clustered. + + + command + text + + The command that is running. Either CLUSTER or VACUUM FULL. + + + + phase + text + + Current processing phase. See or . + + + + cluster_index_relid + bigint + + If the table is being scanned using an index, this is the OID of the + index being used; otherwise, it is zero. + + + + heap_tuples_scanned + bigint + + Number of heap tuples scanned. + This counter only advances when the phase is seq scanning heap, + writing new heap. + + + + heap_tuples_vacuumed + bigint + + Number of heap tuples vacuumed. This counter only advances when the + command is VACUUM FULL and the phase is scanning heap. + + + + +
+ + + CLUSTER phases + + + + Phase + Description + + + + + + initializing + + CLUSTER is preparing to begin scanning the heap. This + phase is expected to be very brief. + + + + seq scanning heap + + CLUSTER is currently scanning heap from the table by + seq scan. + + + + index scanning heap + + CLUSTER is currently scanning heap from the table by + index scan. + + + + sorting tuples + + CLUSTER is currently sorting tuples. + + + + swapping relation files + + CLUSTER is currently swapping old heap and new clustered heap. + + + + rebuilding index + + CLUSTER is currently rebuilding index. + + + + performing final cleanup + + CLUSTER is performing final cleanup. When this phase is + completed, CLUSTER will end. + + + + +
+ + + VACUUM FULL phases + + + + Phase + Description + + + + + + initializing + + VACUUM FULL is preparing to begin scanning the heap. This + phase is expected to be very brief. + + + + seq scanning heap + + VACUUM FULL is currently scanning heap from the table. + + + + swapping relation files + + VACUUM FULL is currently swapping old heap and new vacuumed heap. + + + + rebuilding index + + VACUUM FULL is currently rebuilding index. + + + + performing final cleanup + + VACUUM FULL is performing final cleanup. When this phase is + completed, VACUUM FULL will end. + + + + +
+
--------------533B1B7756EA5C06EE5F72FE--