public inbox for [email protected]help / color / mirror / Atom feed
[PATCH] Report progress of CREATE INDEX operations 12+ messages / 4 participants [nested] [flat]
* [PATCH] Report progress of CREATE INDEX operations @ 2019-01-02 19:14 Alvaro Herrera <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Alvaro Herrera @ 2019-01-02 19:14 UTC (permalink / raw) --- contrib/amcheck/verify_nbtree.c | 2 +- contrib/bloom/blinsert.c | 2 +- contrib/bloom/blutils.c | 1 + doc/src/sgml/indexam.sgml | 13 ++ doc/src/sgml/monitoring.sgml | 227 +++++++++++++++++++++++++- src/backend/access/brin/brin.c | 5 +- src/backend/access/gin/gininsert.c | 2 +- src/backend/access/gin/ginutil.c | 1 + src/backend/access/gist/gist.c | 1 + src/backend/access/gist/gistbuild.c | 2 +- src/backend/access/hash/hash.c | 3 +- src/backend/access/nbtree/nbtree.c | 9 + src/backend/access/nbtree/nbtsort.c | 43 ++++- src/backend/access/nbtree/nbtutils.c | 24 +++ src/backend/access/spgist/spginsert.c | 2 +- src/backend/access/spgist/spgutils.c | 1 + src/backend/catalog/index.c | 134 ++++++++++++++- src/backend/catalog/system_views.sql | 27 +++ src/backend/commands/indexcmds.c | 52 +++++- src/backend/storage/ipc/standby.c | 2 +- src/backend/storage/lmgr/lmgr.c | 46 +++++- src/backend/storage/lmgr/lock.c | 7 +- src/backend/utils/adt/amutils.c | 23 +++ src/backend/utils/adt/pgstatfuncs.c | 2 + src/include/access/amapi.h | 4 + src/include/access/genam.h | 1 + src/include/access/nbtree.h | 11 ++ src/include/catalog/index.h | 2 + src/include/catalog/pg_proc.dat | 10 +- src/include/commands/progress.h | 36 ++++ src/include/pgstat.h | 5 +- src/include/storage/lmgr.h | 4 +- src/include/storage/lock.h | 2 +- 33 files changed, 668 insertions(+), 38 deletions(-) diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c index 964200a7678..99f6ed6bc44 100644 --- a/contrib/amcheck/verify_nbtree.c +++ b/contrib/amcheck/verify_nbtree.c @@ -534,7 +534,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly, RelationGetRelationName(state->rel), RelationGetRelationName(state->heaprel)); - IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, + IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, false, bt_tuple_present_callback, (void *) state, scan); ereport(DEBUG1, diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c index e43fbe0005f..947ee74881f 100644 --- a/contrib/bloom/blinsert.c +++ b/contrib/bloom/blinsert.c @@ -141,7 +141,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo) initCachedPage(&buildstate); /* Do the heap scan */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, bloomBuildCallback, (void *) &buildstate, NULL); diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c index 64583765787..697a37a384b 100644 --- a/contrib/bloom/blutils.c +++ b/contrib/bloom/blutils.c @@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = blcostestimate; amroutine->amoptions = bloptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = blvalidate; amroutine->ambeginscan = blbeginscan; amroutine->amrescan = blrescan; diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml index 05102724ead..fa4a3d0d131 100644 --- a/doc/src/sgml/indexam.sgml +++ b/doc/src/sgml/indexam.sgml @@ -127,6 +127,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; @@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno, <para> <programlisting> +char * +ambuildphasename (int64 phasenum); +</programlisting> + Return the textual name of the given build phase number. + The phase numbers are those reported during an index build via the + <function>pgstat_progress_update_param</function> interface. + The phase names are then exposed in the + <structname>pg_stat_progress_create_index</structname> view. + </para> + + <para> +<programlisting> bool amvalidate (Oid opclassoid); </programlisting> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 0e73cdcddab..47708570f84 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -336,6 +336,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </entry> </row> + <row> + <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry> + <entry>One row for each backend running <command>CREATE INDEX</command>, showing + current progress. + See <xref linkend='create-index-progress-reporting'/>. + </entry> + </row> + <row> <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry> <entry>One row for each backend (including autovacuum worker processes) running @@ -3376,11 +3384,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <para> <productname>PostgreSQL</productname> has the ability to report the progress of - certain commands during command execution. Currently, the only command - which supports progress reporting is <command>VACUUM</command>. This may be - expanded in the future. + certain commands during command execution. Currently, the only commands + which support progress reporting are <command>CREATE INDEX</command> and + <command>VACUUM</command>. This may be expanded in the future. </para> + <sect2 id="create-index-progress-reporting"> + <title>CREATE INDEX Progress Reporting</title> + + <para> + Whenever <command>CREATE INDEX</command> is running, the + <structname>pg_stat_progress_create_index</structname> view will contain + one row for each backend that is currently creating indexes. The tables + below describe the information that will be reported and provide information + about how to interpret it. + </para> + + <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index"> + <title><structname>pg_stat_progress_create_index</structname> View</title> + <tgroup cols="3"> + <thead> + <row> + <entry>Column</entry> + <entry>Type</entry> + <entry>Description</entry> + </row> + </thead> + + <tbody> + <row> + <entry><structfield>pid</structfield></entry> + <entry><type>integer</type></entry> + <entry>Process ID of backend.</entry> + </row> + <row> + <entry><structfield>datid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>datname</structfield></entry> + <entry><type>name</type></entry> + <entry>Name of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>relid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the table on which the index is being created.</entry> + </row> + <row> + <entry><structfield>phase</structfield></entry> + <entry><type>text</type></entry> + <entry> + Current processing phase of index creation. See <xref linkend='create-index-phases'/>. + </entry> + </row> + <row> + <entry><structfield>lockers_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of lockers to wait for, when applicable. + </entry> + </row> + <row> + <entry><structfield>lockers_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of lockers already waited for. + </entry> + </row> + <row> + <entry><structfield>current_locked_pid</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Process ID of the locker currently being waited for. + </entry> + </row> + <row> + <entry><structfield>blocks_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of blocks to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>blocks_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of blocks already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of tuples to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of tuples already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>partitions_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + total number of partitions on which the index is to be created. + </entry> + </row> + <row> + <entry><structfield>partitions_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + number of partitions on which the index has been completed. + </entry> + </row> + </tbody> + </tgroup> + </table> + + <table id="create-index-phases"> + <title>CREATE INDEX phases</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Phase</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry><literal>initializing</literal></entry> + <entry> + <command>CREATE INDEX</command> is preparing to create the index. This + phase is expected to be very brief. + </entry> + </row> + <row> + <entry><literal>waiting for old snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>building index</literal></entry> + <entry> + The index is being built by the access method-specific code. In this phase, + access methods that support progress reporting fill in their own progress data, + and the subphase is indicated in this column. Typically, + <structname>blocks_total</structname> and <structname>blocks_done</structname> + will contain progress data, as well as potentially + <structname>tuples_total</structname> and <structname>tuples_done</structname>. + </entry> + </row> + <row> + <entry><literal>waiting for writer snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially write into the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>validating index scan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching + for tuples that need to be validated. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the index) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>sorting index scan results</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the + previous phase. + </entry> + </row> + <row> + <entry><literal>validating index heapscan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the table + to validate the index tuples collected in the previous two phases. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the table) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>waiting for reader snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. This + phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + </tbody> + </tgroup> + </table> + + </sect2> + <sect2 id="vacuum-progress-reporting"> <title>VACUUM Progress Reporting</title> diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index 8f008dd0080..5d1aff34080 100644 --- a/src/backend/access/brin/brin.c +++ b/src/backend/access/brin/brin.c @@ -111,6 +111,7 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = brincostestimate; amroutine->amoptions = brinoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = brinvalidate; amroutine->ambeginscan = brinbeginscan; amroutine->amrescan = brinrescan; @@ -718,7 +719,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Now scan the relation. No syncscan allowed here because we want the * heap blocks in physical order. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false, brinbuildCallback, (void *) state, NULL); /* process the final batch */ @@ -1234,7 +1235,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel, * by transactions that are still in progress, among other corner cases. */ state->bs_currRangeStart = heapBlk; - IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, + IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, false, heapBlk, scanNumBlks, brinbuildCallback, (void *) state, NULL); diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c index 524ac5be8b5..838de4c1ec3 100644 --- a/src/backend/access/gin/gininsert.c +++ b/src/backend/access/gin/gininsert.c @@ -394,7 +394,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Do the heap scan. We disallow sync scan here because dataPlaceToPage * prefers to receive tuples in TID order. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false, ginBuildCallback, (void *) &buildstate, NULL); /* dump remaining entries to the index */ diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c index afc20232ace..d2360eeafb0 100644 --- a/src/backend/access/gin/ginutil.c +++ b/src/backend/access/gin/ginutil.c @@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gincostestimate; amroutine->amoptions = ginoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = ginvalidate; amroutine->ambeginscan = ginbeginscan; amroutine->amrescan = ginrescan; diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index b75b3a8dacd..0dc36af1e0c 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gistcostestimate; amroutine->amoptions = gistoptions; amroutine->amproperty = gistproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = gistvalidate; amroutine->ambeginscan = gistbeginscan; amroutine->amrescan = gistrescan; diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index bd142a3560d..015f874cc93 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -204,7 +204,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) /* * Do the heap scan. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, gistBuildCallback, (void *) &buildstate, NULL); /* diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c index f1f01a0956d..fc7db5d6a13 100644 --- a/src/backend/access/hash/hash.c +++ b/src/backend/access/hash/hash.c @@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = hashcostestimate; amroutine->amoptions = hashoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = hashvalidate; amroutine->ambeginscan = hashbeginscan; amroutine->amrescan = hashrescan; @@ -159,7 +160,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo) buildstate.heapRel = heap; /* do the heap scan */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, hashbuildCallback, (void *) &buildstate, NULL); if (buildstate.spool) diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index 98917de2efd..f9d32c958f6 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -22,6 +22,7 @@ #include "access/nbtxlog.h" #include "access/relscan.h" #include "access/xlog.h" +#include "commands/progress.h" #include "commands/vacuum.h" #include "miscadmin.h" #include "nodes/execnodes.h" @@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = btcostestimate; amroutine->amoptions = btoptions; amroutine->amproperty = btproperty; + amroutine->ambuildphasename = btbuildphasename; amroutine->amvalidate = btvalidate; amroutine->ambeginscan = btbeginscan; amroutine->amrescan = btrescan; @@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (needLock) UnlockRelationForExtension(rel, ExclusiveLock); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + num_pages); + /* Quit if we've scanned the whole relation */ if (blkno >= num_pages) break; @@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, for (; blkno < num_pages; blkno++) { btvacuumpage(&vstate, blkno, blkno); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blkno); } } diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index dc398e11867..804280385aa 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -65,6 +65,7 @@ #include "access/xlog.h" #include "access/xloginsert.h" #include "catalog/index.h" +#include "commands/progress.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/smgr.h" @@ -288,7 +289,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate, static void _bt_leader_participate_as_worker(BTBuildState *buildstate); static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem); + Sharedsort *sharedsort2, int sortmem, + bool progress); /* @@ -384,6 +386,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Save as primary spool */ buildstate->spool = btspool; + /* Report heap scan phase started */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN); + /* Attempt to launch parallel worker scan when required */ if (indexInfo->ii_ParallelWorkers > 0) _bt_begin_parallel(buildstate, indexInfo->ii_Concurrent, @@ -470,13 +476,17 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Fill spool using either serial or parallel heap scan */ if (!buildstate->btleader) - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true, _bt_build_callback, (void *) buildstate, NULL); else reltuples = _bt_parallel_heapscan(buildstate, &indexInfo->ii_BrokenHotChain); + /* Set the progress target for the next phase */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_TOTAL, + buildstate->indtuples); + /* okay, all heap tuples are spooled */ if (buildstate->spool2 && !buildstate->havedead) { @@ -525,9 +535,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) } #endif /* BTREE_BUILD_STATS */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_1); tuplesort_performsort(btspool->sortstate); if (btspool2) + { + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_2); tuplesort_performsort(btspool2->sortstate); + } wstate.heap = btspool->heap; wstate.index = btspool->index; @@ -543,6 +559,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) wstate.btws_pages_written = 0; wstate.btws_zeropage = NULL; /* until needed */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_LEAF_LOAD); _bt_load(&wstate, btspool, btspool2); } @@ -1078,6 +1096,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index); ScanKey indexScanKey = NULL; SortSupport sortKeys; + long tuples_done = 0L; if (merge) { @@ -1170,6 +1189,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) _bt_buildadd(wstate, state, itup2); itup2 = tuplesort_getindextuple(btspool2->sortstate, true); } + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } pfree(sortKeys); } @@ -1184,6 +1207,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) state = _bt_pagestate(wstate, 0); _bt_buildadd(wstate, state, itup); + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } } @@ -1318,6 +1345,10 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request) btshared->brokenhotchain = false; heap_parallelscan_initialize(&btshared->heapdesc, btspool->heap, snapshot); + /* Report total number of blocks to scan */ + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + btshared->heapdesc.phs_nblocks); + /* * Store shared tuplesort-private state, for which we reserved space. * Then, initialize opaque state using tuplesort routine. @@ -1493,7 +1524,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate) /* Perform work common to all participants */ _bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared, btleader->sharedsort, btleader->sharedsort2, - sortmem); + sortmem, true); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1584,7 +1615,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) /* Perform sorting of spool, and possibly a spool2 */ sortmem = maintenance_work_mem / btshared->scantuplesortstates; _bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort, - sharedsort2, sortmem); + sharedsort2, sortmem, false); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1613,7 +1644,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem) + Sharedsort *sharedsort2, int sortmem, bool progress) { SortCoordinate coordinate; BTBuildState buildstate; @@ -1672,7 +1703,7 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, indexInfo->ii_Concurrent = btshared->isconcurrent; scan = heap_beginscan_parallel(btspool->heap, &btshared->heapdesc); reltuples = IndexBuildHeapScan(btspool->heap, btspool->index, indexInfo, - true, _bt_build_callback, + true, progress, _bt_build_callback, (void *) &buildstate, scan); /* diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c index 2c05fb5e451..295c22b2e5b 100644 --- a/src/backend/access/nbtree/nbtutils.c +++ b/src/backend/access/nbtree/nbtutils.c @@ -20,6 +20,7 @@ #include "access/nbtree.h" #include "access/reloptions.h" #include "access/relscan.h" +#include "commands/progress.h" #include "miscadmin.h" #include "utils/array.h" #include "utils/lsyscache.h" @@ -2082,6 +2083,29 @@ btproperty(Oid index_oid, int attno, } } +/* + * btbuildphasename() -- Return name of index build phase. + */ +char * +btbuildphasename(int64 phasenum) +{ + switch (phasenum) + { + case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE: + return "initializing (1 of 5)"; + case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN: + return "table scan (2 of 5)"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_1: + return "sorting tuples, spool 1 (3 of 5)"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_2: + return "sorting tuples, spool 2 (4 of 5)"; + case PROGRESS_BTREE_PHASE_LEAF_LOAD: + return "btree tuple loading (5 of 5)"; + default: + return NULL; + } +} + /* * _bt_nonkey_truncate() -- create tuple without non-key suffix attributes. * diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c index f428a151385..1bc671c7238 100644 --- a/src/backend/access/spgist/spginsert.c +++ b/src/backend/access/spgist/spginsert.c @@ -142,7 +142,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo) "SP-GiST build temporary context", ALLOCSET_DEFAULT_SIZES); - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, spgistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c index 8e63c1fad25..45472db147b 100644 --- a/src/backend/access/spgist/spgutils.c +++ b/src/backend/access/spgist/spgutils.c @@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = spgcostestimate; amroutine->amoptions = spgoptions; amroutine->amproperty = spgproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = spgvalidate; amroutine->ambeginscan = spgbeginscan; amroutine->amrescan = spgrescan; diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index d16c3d0ea50..4205491ec1f 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -49,8 +49,9 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" -#include "commands/tablecmds.h" #include "commands/event_trigger.h" +#include "commands/progress.h" +#include "commands/tablecmds.h" #include "commands/trigger.h" #include "executor/executor.h" #include "miscadmin.h" @@ -58,6 +59,7 @@ #include "nodes/nodeFuncs.h" #include "optimizer/optimizer.h" #include "parser/parser.h" +#include "pgstat.h" #include "rewrite/rewriteManip.h" #include "storage/bufmgr.h" #include "storage/lmgr.h" @@ -1597,7 +1599,7 @@ index_drop(Oid indexId, bool concurrent) * to acquire an exclusive lock on our table. The lock code will * detect deadlock and error out properly. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * No more predicate locks will be acquired on this index, and we're @@ -1641,7 +1643,7 @@ index_drop(Oid indexId, bool concurrent) * Wait till every transaction that saw the old index state has * finished. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * Re-open relations to allow us to complete our actions. @@ -2291,6 +2293,25 @@ index_build(Relation heapRelation, save_sec_context | SECURITY_RESTRICTED_OPERATION); save_nestlevel = NewGUCNestLevel(); + /* Set up initial progress report status */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_BUILD, + PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE, + 0, 0, 0, 0 + }; + + pgstat_progress_update_multi_param(6, index, val); + } + /* * Call the access method's build procedure */ @@ -2428,13 +2449,14 @@ IndexBuildHeapScan(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, HeapScanDesc scan) { return IndexBuildHeapRangeScan(heapRelation, indexRelation, indexInfo, allow_sync, - false, + false, progress, 0, InvalidBlockNumber, callback, callback_state, scan); } @@ -2455,6 +2477,7 @@ IndexBuildHeapRangeScan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -2476,6 +2499,8 @@ IndexBuildHeapRangeScan(Relation heapRelation, TransactionId OldestXmin; BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; + BlockNumber blocks_done = 0; + BlockNumber previous_blkno = InvalidBlockNumber; /* * sanity checks @@ -2545,6 +2570,13 @@ IndexBuildHeapRangeScan(Relation heapRelation, NULL, /* scan key */ true, /* buffer access strategy OK */ allow_sync); /* syncscan OK? */ + + if (progress) + { + Assert(scan->rs_numblocks == InvalidBlockNumber); + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + scan->rs_nblocks); + } } else { @@ -2592,6 +2624,46 @@ IndexBuildHeapRangeScan(Relation heapRelation, CHECK_FOR_INTERRUPTS(); + /* Report scan progress, if asked to. */ + if (progress && + ((previous_blkno == InvalidBlockNumber) || + (scan->rs_cblock != previous_blkno))) + { + /* we only do progress for full table scans */ + Assert(numblocks == InvalidBlockNumber); + + /* + * Report the number of blocks we've moved forward. + * + * Parallel workers cause the leader process to skip some blocks, + * so we subtract the current block number to the previous one to + * determine how many have been read in total; but be careful when + * we wrap around the last block in the table. + */ + if (scan->rs_cblock > previous_blkno) + blocks_done += scan->rs_cblock - previous_blkno; + else if (previous_blkno == InvalidBlockNumber) + { + /* + * How many blocks have been read since the scan started. + * Should normally be zero. + */ + blocks_done += scan->rs_cblock - + (scan->rs_parallel ? scan->rs_parallel->phs_startblock : + scan->rs_startblock); + Assert(blocks_done == 0); + } + else + { + /* wrapped around */ + blocks_done += scan->rs_nblocks - previous_blkno + scan->rs_cblock; + } + + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blocks_done); + previous_blkno = scan->rs_cblock; + } + /* * When dealing with a HOT-chain of updated tuples, we want to index * the values of the live tuple (if any), but index it under the TID @@ -3133,6 +3205,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) int save_sec_context; int save_nestlevel; + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN, + 0, 0, 0, 0 + }; + pgstat_progress_update_multi_param(5, index, val); + } + /* Open and lock the parent heap relation */ heapRelation = table_open(heapId, ShareUpdateExclusiveLock); /* And the target index relation */ @@ -3163,6 +3250,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) */ ivinfo.index = indexRelation; ivinfo.analyze_only = false; + ivinfo.report_progress = true; /* XXX only for btree? */ ivinfo.estimated_count = true; ivinfo.message_level = DEBUG2; ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples; @@ -3180,15 +3268,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) NULL, false); state.htups = state.itups = state.tups_inserted = 0; + /* ambulkdelete updates progress metrics */ (void) index_bulk_delete(&ivinfo, NULL, validate_index_callback, (void *) &state); /* Execute the sort */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } tuplesort_performsort(state.tuplesort); /* - * Now scan the heap and "merge" it with the index + * Now scan the heap and "merge" it with the index. */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE + }; + + pgstat_progress_update_multi_param(1, index, val); + } validate_index_heapscan(heapRelation, indexRelation, indexInfo, @@ -3291,6 +3403,7 @@ validate_index_heapscan(Relation heapRelation, BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; bool in_index[MaxHeapTuplesPerPage]; + BlockNumber previous_blkno = InvalidBlockNumber; /* state variables for the merge */ ItemPointer indexcursor = NULL; @@ -3330,6 +3443,9 @@ validate_index_heapscan(Relation heapRelation, true, /* buffer access strategy OK */ false); /* syncscan not OK */ + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + scan->rs_nblocks); + /* * Scan all tuples matching the snapshot. */ @@ -3343,6 +3459,14 @@ validate_index_heapscan(Relation heapRelation, state->htups += 1; + if ((previous_blkno == InvalidBlockNumber) || + (scan->rs_cblock != previous_blkno)) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + scan->rs_cblock); + previous_blkno = scan->rs_cblock; + } + /* * As commented in IndexBuildHeapScan, we should index heap-only * tuples under the TIDs of their root tuples; so when we advance onto diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 3e229c693c4..3cbe6f8af2a 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -906,6 +906,33 @@ CREATE VIEW pg_stat_progress_vacuum AS FROM pg_stat_get_progress_info('VACUUM') AS S LEFT JOIN pg_database D ON S.datid = D.oid; +CREATE VIEW pg_stat_progress_create_index AS + SELECT + S.pid AS pid, S.datid AS datid, D.datname AS datname, + S.relid AS relid, + CASE S.param2 WHEN 0 THEN 'initializing (phase 1 of 8)' + WHEN 1 THEN 'waiting for old snapshots (phase 2 of 8)' + WHEN 2 THEN 'building index (3 of 8)' || + COALESCE((': ' || pg_indexam_progress_phasename(S.param1::oid, S.param3)), + '') + WHEN 3 THEN 'waiting for writer snapshots (phase 4 of 8)' + WHEN 4 THEN 'index validation: scan index (phase 5 of 8)' + WHEN 5 THEN 'index validation: sort index scan results (phase 6 of 8)' + WHEN 6 THEN 'index validation: scan heap (phase 7 of 8)' + WHEN 7 THEN 'waiting for reader snapshots (phase 8 of 8)' + END as phase, + S.param4 AS lockers_total, + S.param5 AS lockers_done, + S.param6 AS current_locker_pid, + S.param7 AS blocks_total, + S.param8 AS blocks_done, + S.param9 AS tuples_total, + S.param10 AS tuples_done, + S.param11 AS partitions_total, + S.param12 AS partitions_done + FROM pg_stat_get_progress_info('CREATE INDEX') AS S + LEFT JOIN pg_database D ON S.datid = D.oid; + CREATE VIEW pg_user_mappings AS SELECT U.oid AS umid, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 5dcedc337aa..8f8587ac1e3 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -35,6 +35,7 @@ #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" +#include "commands/progress.h" #include "commands/tablecmds.h" #include "commands/tablespace.h" #include "mb/pg_wchar.h" @@ -46,10 +47,12 @@ #include "parser/parse_func.h" #include "parser/parse_oper.h" #include "partitioning/partdesc.h" +#include "pgstat.h" #include "rewrite/rewriteManip.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/acl.h" #include "utils/builtins.h" #include "utils/fmgroids.h" @@ -368,6 +371,15 @@ DefineIndex(Oid relationId, Snapshot snapshot; int i; + + /* + * Start progress report. If we're building a partition, this was already + * done. + */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX, + relationId); + /* * count key attributes in index */ @@ -584,6 +596,9 @@ DefineIndex(Oid relationId, accessMethodId = accessMethodForm->oid; amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler); + pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID, + accessMethodId); + if (stmt->unique && !amRoutine->amcanunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -864,6 +879,11 @@ DefineIndex(Oid relationId, if (!OidIsValid(indexRelationId)) { table_close(rel, NoLock); + + /* If this is the top-level index, we're done */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -889,6 +909,9 @@ DefineIndex(Oid relationId, TupleDesc parentDesc; Oid *opfamOids; + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL, + nparts); + memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts); parentDesc = CreateTupleDescCopy(RelationGetDescr(rel)); @@ -1039,6 +1062,8 @@ DefineIndex(Oid relationId, skip_build, quiet); } + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, + i + 1); pfree(attmap); } @@ -1073,6 +1098,8 @@ DefineIndex(Oid relationId, * Indexes on partitioned tables are not themselves built, so we're * done here. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); return address; } @@ -1080,6 +1107,11 @@ DefineIndex(Oid relationId, { /* Close the heap and we're done, in the non-concurrent case */ table_close(rel, NoLock); + + /* If this is the top-level index, we're done. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -1131,7 +1163,9 @@ DefineIndex(Oid relationId, * exclusive lock on our table. The lock code will detect deadlock and * error out properly. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_1); + WaitForLockers(heaplocktag, ShareLock, true); /* * At this moment we are sure that there are no transactions with the @@ -1195,7 +1229,9 @@ DefineIndex(Oid relationId, * We once again wait until no transaction can have the table open with * the index marked as read-only for updates. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_2); + WaitForLockers(heaplocktag, ShareLock, true); /* * Now take the "reference snapshot" that will be used by validate_index() @@ -1281,6 +1317,9 @@ DefineIndex(Oid relationId, old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false, PROC_IS_AUTOVACUUM | PROC_IN_VACUUM, &n_old_snapshots); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_3); + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots); for (i = 0; i < n_old_snapshots; i++) { @@ -1316,7 +1355,14 @@ DefineIndex(Oid relationId, } if (VirtualTransactionIdIsValid(old_snapshots[i])) + { + PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId); + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); VirtualXactLock(old_snapshots[i], true); + } + + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1); } /* @@ -1339,6 +1385,8 @@ DefineIndex(Oid relationId, */ UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock); + pgstat_progress_end_command(); + return address; } diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index 4d10e57a803..a0a2b964703 100644 --- a/src/backend/storage/ipc/standby.c +++ b/src/backend/storage/ipc/standby.c @@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag) */ VirtualTransactionId *backends; - backends = GetLockConflicts(&locktag, AccessExclusiveLock); + backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL); ResolveRecoveryConflictWithVirtualXIDs(backends, PROCSIG_RECOVERY_CONFLICT_LOCK); } diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c index e688ba81170..0b04b093782 100644 --- a/src/backend/storage/lmgr/lmgr.c +++ b/src/backend/storage/lmgr/lmgr.c @@ -19,9 +19,12 @@ #include "access/transam.h" #include "access/xact.h" #include "catalog/catalog.h" +#include "commands/progress.h" #include "miscadmin.h" +#include "pgstat.h" #include "storage/lmgr.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/inval.h" @@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg) * after we obtained our initial list of lockers, we will not wait for them. */ void -WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) +WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress) { List *holders = NIL; ListCell *lc; + int total = 0; + int done = 0; /* Done if no locks to wait for */ if (list_length(locktags) == 0) @@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) foreach(lc, locktags) { LOCKTAG *locktag = lfirst(lc); + int count; - holders = lappend(holders, GetLockConflicts(locktag, lockmode)); + holders = lappend(holders, + GetLockConflicts(locktag, lockmode, + progress ? &count : NULL)); + total += count; } + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total); + /* * Note: GetLockConflicts() never reports our own xid, hence we need not * check for that. Also, prepared xacts are not reported, which is fine @@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) while (VirtualTransactionIdIsValid(*lockholders)) { + /* + * If requested, publish who we're going to wait for. This is not + * 100% accurate if they're already gone, but we don't care. + */ + if (progress) + { + PGPROC *holder = BackendIdGetProc(lockholders->backendId); + + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); + } VirtualXactLock(*lockholders, true); lockholders++; + + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done); } } + if (progress) + { + const int index[] = { + PROGRESS_WAITFOR_TOTAL, + PROGRESS_WAITFOR_DONE, + PROGRESS_WAITFOR_CURRENT_PID + }; + const int64 values[] = { + 0, 0, 0 + }; + pgstat_progress_update_multi_param(3, index, values); + } list_free_deep(holders); } @@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) * Same as WaitForLockersMultiple, for a single lock tag. */ void -WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode) +WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress) { List *l; l = list_make1(&heaplocktag); - WaitForLockersMultiple(l, lockmode); + WaitForLockersMultiple(l, lockmode, progress); list_free(l); } diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c index 3bb5ce350aa..58ba90d0646 100644 --- a/src/backend/storage/lmgr/lock.c +++ b/src/backend/storage/lmgr/lock.c @@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * xacts merely awaiting such a lock are NOT reported. * * The result array is palloc'd and is terminated with an invalid VXID. + * *countp, if not null, is updated to the number of items set. * * Of course, the result could be out of date by the time it's returned, * so use of this function has to be thought about carefully. @@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * uses of the result. */ VirtualTransactionId * -GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) +GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp) { static VirtualTransactionId *vxids; LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid; @@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) LWLockRelease(partitionLock); vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } @@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c index 060ffe501ec..e81d6cc0562 100644 --- a/src/backend/utils/adt/amutils.c +++ b/src/backend/utils/adt/amutils.c @@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS) return indexam_property(fcinfo, propname, InvalidOid, relid, attno); } + +/* + * Return the name of the given phase, as used for progress reporting by the + * given AM. + */ +Datum +pg_indexam_progress_phasename(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + int32 phasenum = PG_GETARG_INT32(1); + IndexAmRoutine *routine; + char *name; + + routine = GetIndexAmRoutineByAmId(amoid, true); + if (routine == NULL || !routine->ambuildphasename) + PG_RETURN_NULL(); + + name = routine->ambuildphasename(phasenum); + if (!name) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(CStringGetTextDatum(name)); +} diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 69f72657792..05f9da6a4ad 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS) /* Translate command name into command type code. */ if (pg_strcasecmp(cmd, "VACUUM") == 0) cmdtype = PROGRESS_COMMAND_VACUUM; + else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0) + cmdtype = PROGRESS_COMMAND_CREATE_INDEX; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index 653ddc976ba..09a7404267c 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +/* name of phase as used in progress reporting */ +typedef char *(*ambuildphasename_function) (int64 phasenum); + /* validate definition of an opclass for this AM */ typedef bool (*amvalidate_function) (Oid opclassoid); @@ -213,6 +216,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; diff --git a/src/include/access/genam.h b/src/include/access/genam.h index c4aba39496f..f77d9eea8e8 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo { Relation index; /* the index being vacuumed */ bool analyze_only; /* ANALYZE (without any actual vacuum) */ + bool report_progress; /* emit progress.h status reports */ bool estimated_count; /* num_heap_tuples is an estimate */ int message_level; /* ereport level for progress messages */ double num_heap_tuples; /* tuples remaining in heap */ diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h index 4fb92d60a12..18a6838ce56 100644 --- a/src/include/access/nbtree.h +++ b/src/include/access/nbtree.h @@ -488,6 +488,16 @@ typedef BTScanOpaqueData *BTScanOpaque; #define SK_BT_DESC (INDOPTION_DESC << SK_BT_INDOPTION_SHIFT) #define SK_BT_NULLS_FIRST (INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT) +/* + * Constant definition for progress reporting. Phase numbers must match + * btbuildphasename. + */ +/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */ +#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN 2 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_1 3 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_2 4 +#define PROGRESS_BTREE_PHASE_LEAF_LOAD 5 + /* * external entry points for btree, in nbtree.c */ @@ -600,6 +610,7 @@ extern bytea *btoptions(Datum reloptions, bool validate); extern bool btproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +extern char *btbuildphasename(int64 phasenum); extern IndexTuple _bt_nonkey_truncate(Relation rel, IndexTuple itup); extern bool _bt_check_natts(Relation rel, Page page, OffsetNumber offnum); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 330c481a8b7..fb713d50f96 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -114,6 +114,7 @@ extern double IndexBuildHeapScan(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, struct HeapScanDescData *scan); @@ -122,6 +123,7 @@ extern double IndexBuildHeapRangeScan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber end_blockno, IndexBuildCallback callback, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index a4e173b4846..32e31c2d3f1 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -917,6 +917,10 @@ proname => 'pg_index_column_has_property', provolatile => 's', prorettype => 'bool', proargtypes => 'regclass int4 text', prosrc => 'pg_index_column_has_property' }, +{ oid => '676', descr => 'return name of given index build phase', + proname => 'pg_indexam_progress_phasename', provolatile => 'i', + prorettype => 'text', proargtypes => 'oid int8', + prosrc => 'pg_indexam_progress_phasename' }, { oid => '339', proname => 'poly_same', prorettype => 'bool', @@ -5079,9 +5083,9 @@ proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't', provolatile => 's', proparallel => 'r', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', - proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}', + proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}', prosrc => 'pg_stat_get_progress_info' }, { oid => '3099', descr => 'statistics: information about currently active replication', diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h index 9858b36a383..56bd6125594 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -34,4 +34,40 @@ #define PROGRESS_VACUUM_PHASE_TRUNCATE 5 #define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6 + +/* Progress parameters for CREATE INDEX */ +#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID 0 +#define PROGRESS_CREATEIDX_PHASE 1 /* AM-agnostic phase # */ +#define PROGRESS_CREATEIDX_SUBPHASE 2 /* phase # filled by AM */ +/* 3, 4 and 5 reserved for "waitfor" metrics */ +/* 6 and 7 reserved for "block number" metrics */ +#define PROGRESS_CREATEIDX_TUPLES_TOTAL 8 +#define PROGRESS_CREATEIDX_TUPLES_DONE 9 +#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL 10 +#define PROGRESS_CREATEIDX_PARTITIONS_DONE 11 + +/* Phases of CREATE INDEX */ +#define PROGRESS_CREATEIDX_PHASE_WAIT_1 1 +#define PROGRESS_CREATEIDX_PHASE_BUILD 2 +#define PROGRESS_CREATEIDX_PHASE_WAIT_2 3 +#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN 4 +#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN 5 +#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE 6 +#define PROGRESS_CREATEIDX_PHASE_WAIT_3 7 + +/* + * Subphases of CREATE INDEX, for index_build. + */ +#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE 1 +/* Additional phases are defined by each AM */ + +/* Lock holder wait counts */ +#define PROGRESS_WAITFOR_TOTAL 3 +#define PROGRESS_WAITFOR_DONE 4 +#define PROGRESS_WAITFOR_CURRENT_PID 5 + +/* Block numbers in a generic relation scan */ +#define PROGRESS_SCAN_BLOCKS_TOTAL 6 +#define PROGRESS_SCAN_BLOCKS_DONE 7 + #endif diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 88a75fb798e..f931ead1c27 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -934,10 +934,11 @@ typedef enum typedef enum ProgressCommandType { PROGRESS_COMMAND_INVALID, - PROGRESS_COMMAND_VACUUM + PROGRESS_COMMAND_VACUUM, + PROGRESS_COMMAND_CREATE_INDEX } ProgressCommandType; -#define PGSTAT_NUM_PROGRESS_PARAM 10 +#define PGSTAT_NUM_PROGRESS_PARAM 12 /* ---------- * Shared-memory data structures diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h index 3d705faba5c..4f2872de35f 100644 --- a/src/include/storage/lmgr.h +++ b/src/include/storage/lmgr.h @@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel, extern bool ConditionalXactLockTableWait(TransactionId xid); /* Lock VXIDs, specified by conflicting locktags */ -extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode); -extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode); +extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress); +extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress); /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */ extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid); diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h index 16b927cb801..e117b391774 100644 --- a/src/include/storage/lock.h +++ b/src/include/storage/lock.h @@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode); extern bool LockHasWaiters(const LOCKTAG *locktag, LOCKMODE lockmode, bool sessionLock); extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag, - LOCKMODE lockmode); + LOCKMODE lockmode, int *countp); extern void AtPrepare_Locks(void); extern void PostPrepare_Locks(TransactionId xid); extern int LockCheckConflicts(LockMethod lockMethodTable, -- 2.17.1 --a8Wt8u1KmwUX3Y2C-- ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v6 2/3] Report progress of CREATE INDEX operations @ 2019-01-02 19:14 Alvaro Herrera <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Alvaro Herrera @ 2019-01-02 19:14 UTC (permalink / raw) --- contrib/amcheck/verify_nbtree.c | 2 +- contrib/bloom/blinsert.c | 2 +- contrib/bloom/blutils.c | 1 + doc/src/sgml/indexam.sgml | 13 ++ doc/src/sgml/monitoring.sgml | 224 +++++++++++++++++++++++++- src/backend/access/brin/brin.c | 5 +- src/backend/access/gin/gininsert.c | 2 +- src/backend/access/gin/ginutil.c | 1 + src/backend/access/gist/gist.c | 1 + src/backend/access/gist/gistbuild.c | 2 +- src/backend/access/hash/hash.c | 3 +- src/backend/access/nbtree/nbtree.c | 9 ++ src/backend/access/nbtree/nbtsort.c | 58 ++++++- src/backend/access/nbtree/nbtutils.c | 24 +++ src/backend/access/spgist/spginsert.c | 2 +- src/backend/access/spgist/spgutils.c | 1 + src/backend/catalog/index.c | 118 +++++++++++++- src/backend/catalog/system_views.sql | 27 ++++ src/backend/commands/indexcmds.c | 52 +++++- src/backend/storage/ipc/standby.c | 2 +- src/backend/storage/lmgr/lmgr.c | 46 +++++- src/backend/storage/lmgr/lock.c | 7 +- src/backend/utils/adt/amutils.c | 23 +++ src/backend/utils/adt/pgstatfuncs.c | 2 + src/include/access/amapi.h | 4 + src/include/access/genam.h | 1 + src/include/access/nbtree.h | 11 ++ src/include/catalog/index.h | 2 + src/include/catalog/pg_proc.dat | 10 +- src/include/commands/progress.h | 35 ++++ src/include/pgstat.h | 5 +- src/include/storage/lmgr.h | 4 +- src/include/storage/lock.h | 2 +- src/test/regress/expected/rules.out | 30 +++- 34 files changed, 693 insertions(+), 38 deletions(-) diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c index 6ae3bca9536..ebad178476f 100644 --- a/contrib/amcheck/verify_nbtree.c +++ b/contrib/amcheck/verify_nbtree.c @@ -565,7 +565,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace, RelationGetRelationName(state->rel), RelationGetRelationName(state->heaprel)); - IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, + IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, false, bt_tuple_present_callback, (void *) state, scan); ereport(DEBUG1, diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c index e43fbe0005f..947ee74881f 100644 --- a/contrib/bloom/blinsert.c +++ b/contrib/bloom/blinsert.c @@ -141,7 +141,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo) initCachedPage(&buildstate); /* Do the heap scan */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, bloomBuildCallback, (void *) &buildstate, NULL); diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c index d078dfbd469..ee3bd562748 100644 --- a/contrib/bloom/blutils.c +++ b/contrib/bloom/blutils.c @@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = blcostestimate; amroutine->amoptions = bloptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = blvalidate; amroutine->ambeginscan = blbeginscan; amroutine->amrescan = blrescan; diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml index 05102724ead..fa4a3d0d131 100644 --- a/doc/src/sgml/indexam.sgml +++ b/doc/src/sgml/indexam.sgml @@ -127,6 +127,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; @@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno, <para> <programlisting> +char * +ambuildphasename (int64 phasenum); +</programlisting> + Return the textual name of the given build phase number. + The phase numbers are those reported during an index build via the + <function>pgstat_progress_update_param</function> interface. + The phase names are then exposed in the + <structname>pg_stat_progress_create_index</structname> view. + </para> + + <para> +<programlisting> bool amvalidate (Oid opclassoid); </programlisting> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index f1df14bdea8..8f5bbbb79e5 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -336,6 +336,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </entry> </row> + <row> + <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry> + <entry>One row for each backend running <command>CREATE INDEX</command>, showing + current progress. + See <xref linkend='create-index-progress-reporting'/>. + </entry> + </row> + <row> <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry> <entry>One row for each backend (including autovacuum worker processes) running @@ -3403,10 +3411,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <para> <productname>PostgreSQL</productname> has the ability to report the progress of certain commands during command execution. Currently, the only commands - which support progress reporting are <command>VACUUM</command> and + which support progress reporting are <command>CREATE INDEX</command>, + <command>VACUUM</command> and <command>CLUSTER</command>. This may be expanded in the future. </para> + <sect2 id="create-index-progress-reporting"> + <title>CREATE INDEX Progress Reporting</title> + + <para> + Whenever <command>CREATE INDEX</command> is running, the + <structname>pg_stat_progress_create_index</structname> view will contain + one row for each backend that is currently creating indexes. The tables + below describe the information that will be reported and provide information + about how to interpret it. + </para> + + <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index"> + <title><structname>pg_stat_progress_create_index</structname> View</title> + <tgroup cols="3"> + <thead> + <row> + <entry>Column</entry> + <entry>Type</entry> + <entry>Description</entry> + </row> + </thead> + + <tbody> + <row> + <entry><structfield>pid</structfield></entry> + <entry><type>integer</type></entry> + <entry>Process ID of backend.</entry> + </row> + <row> + <entry><structfield>datid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>datname</structfield></entry> + <entry><type>name</type></entry> + <entry>Name of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>relid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the table on which the index is being created.</entry> + </row> + <row> + <entry><structfield>phase</structfield></entry> + <entry><type>text</type></entry> + <entry> + Current processing phase of index creation. See <xref linkend='create-index-phases'/>. + </entry> + </row> + <row> + <entry><structfield>lockers_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of lockers to wait for, when applicable. + </entry> + </row> + <row> + <entry><structfield>lockers_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of lockers already waited for. + </entry> + </row> + <row> + <entry><structfield>current_locked_pid</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Process ID of the locker currently being waited for. + </entry> + </row> + <row> + <entry><structfield>blocks_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of blocks to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>blocks_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of blocks already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of tuples to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of tuples already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>partitions_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + total number of partitions on which the index is to be created. + </entry> + </row> + <row> + <entry><structfield>partitions_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + number of partitions on which the index has been completed. + </entry> + </row> + </tbody> + </tgroup> + </table> + + <table id="create-index-phases"> + <title>CREATE INDEX phases</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Phase</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry><literal>initializing</literal></entry> + <entry> + <command>CREATE INDEX</command> is preparing to create the index. This + phase is expected to be very brief. + </entry> + </row> + <row> + <entry><literal>waiting for old snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>building index</literal></entry> + <entry> + The index is being built by the access method-specific code. In this phase, + access methods that support progress reporting fill in their own progress data, + and the subphase is indicated in this column. Typically, + <structname>blocks_total</structname> and <structname>blocks_done</structname> + will contain progress data, as well as potentially + <structname>tuples_total</structname> and <structname>tuples_done</structname>. + </entry> + </row> + <row> + <entry><literal>waiting for writer snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially write into the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>validating index scan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching + for tuples that need to be validated. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the index) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>sorting index scan results</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the + previous phase. + </entry> + </row> + <row> + <entry><literal>validating index heapscan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the table + to validate the index tuples collected in the previous two phases. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the table) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>waiting for reader snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. This + phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + </tbody> + </tgroup> + </table> + + </sect2> + <sect2 id="vacuum-progress-reporting"> <title>VACUUM Progress Reporting</title> diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index 8f008dd0080..5d1aff34080 100644 --- a/src/backend/access/brin/brin.c +++ b/src/backend/access/brin/brin.c @@ -111,6 +111,7 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = brincostestimate; amroutine->amoptions = brinoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = brinvalidate; amroutine->ambeginscan = brinbeginscan; amroutine->amrescan = brinrescan; @@ -718,7 +719,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Now scan the relation. No syncscan allowed here because we want the * heap blocks in physical order. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false, brinbuildCallback, (void *) state, NULL); /* process the final batch */ @@ -1234,7 +1235,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel, * by transactions that are still in progress, among other corner cases. */ state->bs_currRangeStart = heapBlk; - IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, + IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, false, heapBlk, scanNumBlks, brinbuildCallback, (void *) state, NULL); diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c index 524ac5be8b5..838de4c1ec3 100644 --- a/src/backend/access/gin/gininsert.c +++ b/src/backend/access/gin/gininsert.c @@ -394,7 +394,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Do the heap scan. We disallow sync scan here because dataPlaceToPage * prefers to receive tuples in TID order. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false, ginBuildCallback, (void *) &buildstate, NULL); /* dump remaining entries to the index */ diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c index afc20232ace..d2360eeafb0 100644 --- a/src/backend/access/gin/ginutil.c +++ b/src/backend/access/gin/ginutil.c @@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gincostestimate; amroutine->amoptions = ginoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = ginvalidate; amroutine->ambeginscan = ginbeginscan; amroutine->amrescan = ginrescan; diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index a746e911f37..5c08b611c8d 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gistcostestimate; amroutine->amoptions = gistoptions; amroutine->amproperty = gistproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = gistvalidate; amroutine->ambeginscan = gistbeginscan; amroutine->amrescan = gistrescan; diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index bd142a3560d..015f874cc93 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -204,7 +204,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) /* * Do the heap scan. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, gistBuildCallback, (void *) &buildstate, NULL); /* diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c index f1f01a0956d..fc7db5d6a13 100644 --- a/src/backend/access/hash/hash.c +++ b/src/backend/access/hash/hash.c @@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = hashcostestimate; amroutine->amoptions = hashoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = hashvalidate; amroutine->ambeginscan = hashbeginscan; amroutine->amrescan = hashrescan; @@ -159,7 +160,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo) buildstate.heapRel = heap; /* do the heap scan */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, hashbuildCallback, (void *) &buildstate, NULL); if (buildstate.spool) diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index ac6f1eb3423..7370379c6a1 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -22,6 +22,7 @@ #include "access/nbtxlog.h" #include "access/relscan.h" #include "access/xlog.h" +#include "commands/progress.h" #include "commands/vacuum.h" #include "miscadmin.h" #include "nodes/execnodes.h" @@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = btcostestimate; amroutine->amoptions = btoptions; amroutine->amproperty = btproperty; + amroutine->ambuildphasename = btbuildphasename; amroutine->amvalidate = btvalidate; amroutine->ambeginscan = btbeginscan; amroutine->amrescan = btrescan; @@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (needLock) UnlockRelationForExtension(rel, ExclusiveLock); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + num_pages); + /* Quit if we've scanned the whole relation */ if (blkno >= num_pages) break; @@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, for (; blkno < num_pages; blkno++) { btvacuumpage(&vstate, blkno, blkno); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blkno); } } diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index 2762a2d5485..a47e0de437c 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -66,6 +66,7 @@ #include "access/xlog.h" #include "access/xloginsert.h" #include "catalog/index.h" +#include "commands/progress.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/smgr.h" @@ -298,7 +299,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate, static void _bt_leader_participate_as_worker(BTBuildState *buildstate); static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem); + Sharedsort *sharedsort2, int sortmem, + bool progress); /* @@ -394,6 +396,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Save as primary spool */ buildstate->spool = btspool; + /* Report heap scan phase started */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN); + /* Attempt to launch parallel worker scan when required */ if (indexInfo->ii_ParallelWorkers > 0) _bt_begin_parallel(buildstate, indexInfo->ii_Concurrent, @@ -480,13 +486,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Fill spool using either serial or parallel heap scan */ if (!buildstate->btleader) - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true, _bt_build_callback, (void *) buildstate, NULL); else reltuples = _bt_parallel_heapscan(buildstate, &indexInfo->ii_BrokenHotChain); + /* + * Set the progress target for the next phase. Reset the block number + * values set by IndexBuildHeapScan + */ + { + const int index[] = { + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE + }; + const int64 val[] = { + buildstate->indtuples, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } + /* okay, all heap tuples are spooled */ if (buildstate->spool2 && !buildstate->havedead) { @@ -535,9 +559,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) } #endif /* BTREE_BUILD_STATS */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_1); tuplesort_performsort(btspool->sortstate); if (btspool2) + { + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_2); tuplesort_performsort(btspool2->sortstate); + } wstate.heap = btspool->heap; wstate.index = btspool->index; @@ -554,6 +584,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) wstate.btws_pages_written = 0; wstate.btws_zeropage = NULL; /* until needed */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_LEAF_LOAD); _bt_load(&wstate, btspool, btspool2); } @@ -1098,6 +1130,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) int i, keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index); SortSupport sortKeys; + long tuples_done = 0L; if (merge) { @@ -1202,6 +1235,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) _bt_buildadd(wstate, state, itup2); itup2 = tuplesort_getindextuple(btspool2->sortstate, true); } + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } pfree(sortKeys); } @@ -1216,6 +1253,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) state = _bt_pagestate(wstate, 0); _bt_buildadd(wstate, state, itup); + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } } @@ -1352,6 +1393,11 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request) ParallelTableScanFromBTShared(btshared), snapshot); + /* Report total number of blocks to scan */ + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + ((ParallelBlockTableScanDescData *) + ParallelTableScanFromBTShared(btshared))->phs_nblocks); + /* * Store shared tuplesort-private state, for which we reserved space. * Then, initialize opaque state using tuplesort routine. @@ -1528,7 +1574,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate) /* Perform work common to all participants */ _bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared, btleader->sharedsort, btleader->sharedsort2, - sortmem); + sortmem, true); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1619,7 +1665,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) /* Perform sorting of spool, and possibly a spool2 */ sortmem = maintenance_work_mem / btshared->scantuplesortstates; _bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort, - sharedsort2, sortmem); + sharedsort2, sortmem, false); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1648,7 +1694,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem) + Sharedsort *sharedsort2, int sortmem, bool progress) { SortCoordinate coordinate; BTBuildState buildstate; @@ -1708,7 +1754,7 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, scan = table_beginscan_parallel(btspool->heap, ParallelTableScanFromBTShared(btshared)); reltuples = IndexBuildHeapScan(btspool->heap, btspool->index, indexInfo, - true, _bt_build_callback, + true, progress, _bt_build_callback, (void *) &buildstate, scan); /* diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c index 92b8b5f134d..9e8a7bd3d9a 100644 --- a/src/backend/access/nbtree/nbtutils.c +++ b/src/backend/access/nbtree/nbtutils.c @@ -20,6 +20,7 @@ #include "access/nbtree.h" #include "access/reloptions.h" #include "access/relscan.h" +#include "commands/progress.h" #include "miscadmin.h" #include "utils/array.h" #include "utils/datum.h" @@ -2048,6 +2049,29 @@ btproperty(Oid index_oid, int attno, } } +/* + * btbuildphasename() -- Return name of index build phase. + */ +char * +btbuildphasename(int64 phasenum) +{ + switch (phasenum) + { + case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE: + return "initializing"; + case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN: + return "table scan"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_1: + return "sorting tuples, spool 1"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_2: + return "sorting tuples, spool 2"; + case PROGRESS_BTREE_PHASE_LEAF_LOAD: + return "btree tuple loading"; + default: + return NULL; + } +} + /* * _bt_truncate() -- create tuple without unneeded suffix attributes. * diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c index f428a151385..1bc671c7238 100644 --- a/src/backend/access/spgist/spginsert.c +++ b/src/backend/access/spgist/spginsert.c @@ -142,7 +142,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo) "SP-GiST build temporary context", ALLOCSET_DEFAULT_SIZES); - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, spgistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c index 8e63c1fad25..45472db147b 100644 --- a/src/backend/access/spgist/spgutils.c +++ b/src/backend/access/spgist/spgutils.c @@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = spgcostestimate; amroutine->amoptions = spgoptions; amroutine->amproperty = spgproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = spgvalidate; amroutine->ambeginscan = spgbeginscan; amroutine->amrescan = spgrescan; diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index d2e284f6de6..7778341bf51 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -50,9 +50,9 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" -#include "commands/tablecmds.h" #include "commands/event_trigger.h" #include "commands/progress.h" +#include "commands/tablecmds.h" #include "commands/trigger.h" #include "executor/executor.h" #include "miscadmin.h" @@ -1596,7 +1596,7 @@ index_drop(Oid indexId, bool concurrent) * to acquire an exclusive lock on our table. The lock code will * detect deadlock and error out properly. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * No more predicate locks will be acquired on this index, and we're @@ -1640,7 +1640,7 @@ index_drop(Oid indexId, bool concurrent) * Wait till every transaction that saw the old index state has * finished. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * Re-open relations to allow us to complete our actions. @@ -2290,6 +2290,25 @@ index_build(Relation heapRelation, save_sec_context | SECURITY_RESTRICTED_OPERATION); save_nestlevel = NewGUCNestLevel(); + /* Set up initial progress report status */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_BUILD, + PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE, + 0, 0, 0, 0 + }; + + pgstat_progress_update_multi_param(6, index, val); + } + /* * Call the access method's build procedure */ @@ -2416,6 +2435,12 @@ index_build(Relation heapRelation, * do so here because the AM might reject some of the tuples for its own * reasons, such as being unable to store NULLs. * + * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go + * along. Also, if that flag is true and a scan descriptor is not passed (ie. + * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is + * updated at the beginning. For parallel scans, caller is expected to have + * set the total number of blocks prior to calling this function. + * * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect * any potentially broken HOT chains. Currently, we set this if there are * any RECENTLY_DEAD or DELETE_IN_PROGRESS entries in a HOT chain, without @@ -2427,13 +2452,14 @@ IndexBuildHeapScan(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, TableScanDesc scan) { return IndexBuildHeapRangeScan(heapRelation, indexRelation, indexInfo, allow_sync, - false, + false, progress, 0, InvalidBlockNumber, callback, callback_state, scan); } @@ -2444,6 +2470,13 @@ IndexBuildHeapScan(Relation heapRelation, * passing InvalidBlockNumber as numblocks. Note that restricting the range * to scan cannot be done when requesting syncscan. * + * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go + * along. Also, if that flag is true and a scan descriptor is not passed (ie. + * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is + * updated at the beginning. For parallel scans, caller is expected to have + * set the total number of blocks prior to calling this function. Note this + * only works for full-relation scans. + * * When "anyvisible" mode is requested, all tuples visible to any transaction * are indexed and counted as live, including those inserted or deleted by * transactions that are still in progress. @@ -2454,6 +2487,7 @@ IndexBuildHeapRangeScan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -2476,6 +2510,7 @@ IndexBuildHeapRangeScan(Relation heapRelation, TransactionId OldestXmin; BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; + BlockNumber previous_blkno = InvalidBlockNumber; /* * sanity checks @@ -2544,6 +2579,14 @@ IndexBuildHeapRangeScan(Relation heapRelation, NULL, /* scan key */ true, /* buffer access strategy OK */ allow_sync); /* syncscan OK? */ + + if (progress) + { + hscan = (HeapScanDesc) scan; + Assert(hscan->rs_numblocks == InvalidBlockNumber); + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + hscan->rs_nblocks); + } } else { @@ -2593,6 +2636,19 @@ IndexBuildHeapRangeScan(Relation heapRelation, CHECK_FOR_INTERRUPTS(); + /* Report scan progress, if asked to. */ + if (progress) + { + BlockNumber blocks_done = heapscan_get_blocks_done(scan); + + if (blocks_done != previous_blkno) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blocks_done); + previous_blkno = blocks_done; + } + } + /* * When dealing with a HOT-chain of updated tuples, we want to index * the values of the live tuple (if any), but index it under the TID @@ -3129,6 +3185,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) int save_sec_context; int save_nestlevel; + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN, + 0, 0, 0, 0 + }; + pgstat_progress_update_multi_param(5, index, val); + } + /* Open and lock the parent heap relation */ heapRelation = table_open(heapId, ShareUpdateExclusiveLock); /* And the target index relation */ @@ -3159,6 +3230,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) */ ivinfo.index = indexRelation; ivinfo.analyze_only = false; + ivinfo.report_progress = true; /* XXX only for btree? */ ivinfo.estimated_count = true; ivinfo.message_level = DEBUG2; ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples; @@ -3176,15 +3248,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) NULL, false); state.htups = state.itups = state.tups_inserted = 0; + /* ambulkdelete updates progress metrics */ (void) index_bulk_delete(&ivinfo, NULL, validate_index_callback, (void *) &state); /* Execute the sort */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } tuplesort_performsort(state.tuplesort); /* - * Now scan the heap and "merge" it with the index + * Now scan the heap and "merge" it with the index. */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE + }; + + pgstat_progress_update_multi_param(1, index, val); + } validate_index_heapscan(heapRelation, indexRelation, indexInfo, @@ -3288,6 +3384,7 @@ validate_index_heapscan(Relation heapRelation, BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; bool in_index[MaxHeapTuplesPerPage]; + BlockNumber previous_blkno = InvalidBlockNumber; /* state variables for the merge */ ItemPointer indexcursor = NULL; @@ -3328,6 +3425,9 @@ validate_index_heapscan(Relation heapRelation, false); /* syncscan not OK */ hscan = (HeapScanDesc) scan; + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + hscan->rs_nblocks); + /* * Scan all tuples matching the snapshot. */ @@ -3341,6 +3441,14 @@ validate_index_heapscan(Relation heapRelation, state->htups += 1; + if ((previous_blkno == InvalidBlockNumber) || + (hscan->rs_cblock != previous_blkno)) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + hscan->rs_cblock); + previous_blkno = hscan->rs_cblock; + } + /* * As commented in IndexBuildHeapScan, we should index heap-only * tuples under the TIDs of their root tuples; so when we advance onto diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index b89df70653e..b7c040d5284 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -934,6 +934,33 @@ CREATE VIEW pg_stat_progress_cluster AS FROM pg_stat_get_progress_info('CLUSTER') AS S LEFT JOIN pg_database D ON S.datid = D.oid; +CREATE VIEW pg_stat_progress_create_index AS + SELECT + S.pid AS pid, S.datid AS datid, D.datname AS datname, + S.relid AS relid, + CASE S.param2 WHEN 0 THEN 'initializing' + WHEN 1 THEN 'waiting for old snapshots' + WHEN 2 THEN 'building index' || + COALESCE((': ' || pg_indexam_progress_phasename(S.param1::oid, S.param3)), + '') + WHEN 3 THEN 'waiting for writer snapshots' + WHEN 4 THEN 'index validation: scan index' + WHEN 5 THEN 'index validation: sort index scan results' + WHEN 6 THEN 'index validation: scan heap' + WHEN 7 THEN 'waiting for reader snapshots' + END as phase, + S.param4 AS lockers_total, + S.param5 AS lockers_done, + S.param6 AS current_locker_pid, + S.param7 AS blocks_total, + S.param8 AS blocks_done, + S.param9 AS tuples_total, + S.param10 AS tuples_done, + S.param11 AS partitions_total, + S.param12 AS partitions_done + FROM pg_stat_get_progress_info('CREATE INDEX') AS S + LEFT JOIN pg_database D ON S.datid = D.oid; + CREATE VIEW pg_user_mappings AS SELECT U.oid AS umid, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index c3a53d81aab..9fad6705e45 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -36,6 +36,7 @@ #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" +#include "commands/progress.h" #include "commands/tablecmds.h" #include "commands/tablespace.h" #include "mb/pg_wchar.h" @@ -47,10 +48,12 @@ #include "parser/parse_func.h" #include "parser/parse_oper.h" #include "partitioning/partdesc.h" +#include "pgstat.h" #include "rewrite/rewriteManip.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/acl.h" #include "utils/builtins.h" #include "utils/fmgroids.h" @@ -369,6 +372,15 @@ DefineIndex(Oid relationId, Snapshot snapshot; int i; + + /* + * Start progress report. If we're building a partition, this was already + * done. + */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX, + relationId); + /* * count key attributes in index */ @@ -585,6 +597,9 @@ DefineIndex(Oid relationId, accessMethodId = accessMethodForm->oid; amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler); + pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID, + accessMethodId); + if (stmt->unique && !amRoutine->amcanunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -865,6 +880,11 @@ DefineIndex(Oid relationId, if (!OidIsValid(indexRelationId)) { table_close(rel, NoLock); + + /* If this is the top-level index, we're done */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -890,6 +910,9 @@ DefineIndex(Oid relationId, TupleDesc parentDesc; Oid *opfamOids; + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL, + nparts); + memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts); parentDesc = CreateTupleDescCopy(RelationGetDescr(rel)); @@ -1040,6 +1063,8 @@ DefineIndex(Oid relationId, skip_build, quiet); } + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, + i + 1); pfree(attmap); } @@ -1074,6 +1099,8 @@ DefineIndex(Oid relationId, * Indexes on partitioned tables are not themselves built, so we're * done here. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); return address; } @@ -1081,6 +1108,11 @@ DefineIndex(Oid relationId, { /* Close the heap and we're done, in the non-concurrent case */ table_close(rel, NoLock); + + /* If this is the top-level index, we're done. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -1132,7 +1164,9 @@ DefineIndex(Oid relationId, * exclusive lock on our table. The lock code will detect deadlock and * error out properly. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_1); + WaitForLockers(heaplocktag, ShareLock, true); /* * At this moment we are sure that there are no transactions with the @@ -1196,7 +1230,9 @@ DefineIndex(Oid relationId, * We once again wait until no transaction can have the table open with * the index marked as read-only for updates. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_2); + WaitForLockers(heaplocktag, ShareLock, true); /* * Now take the "reference snapshot" that will be used by validate_index() @@ -1282,6 +1318,9 @@ DefineIndex(Oid relationId, old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false, PROC_IS_AUTOVACUUM | PROC_IN_VACUUM, &n_old_snapshots); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_3); + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots); for (i = 0; i < n_old_snapshots; i++) { @@ -1317,7 +1356,14 @@ DefineIndex(Oid relationId, } if (VirtualTransactionIdIsValid(old_snapshots[i])) + { + PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId); + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); VirtualXactLock(old_snapshots[i], true); + } + + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1); } /* @@ -1340,6 +1386,8 @@ DefineIndex(Oid relationId, */ UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock); + pgstat_progress_end_command(); + return address; } diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index 4d10e57a803..a0a2b964703 100644 --- a/src/backend/storage/ipc/standby.c +++ b/src/backend/storage/ipc/standby.c @@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag) */ VirtualTransactionId *backends; - backends = GetLockConflicts(&locktag, AccessExclusiveLock); + backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL); ResolveRecoveryConflictWithVirtualXIDs(backends, PROCSIG_RECOVERY_CONFLICT_LOCK); } diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c index e688ba81170..0b04b093782 100644 --- a/src/backend/storage/lmgr/lmgr.c +++ b/src/backend/storage/lmgr/lmgr.c @@ -19,9 +19,12 @@ #include "access/transam.h" #include "access/xact.h" #include "catalog/catalog.h" +#include "commands/progress.h" #include "miscadmin.h" +#include "pgstat.h" #include "storage/lmgr.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/inval.h" @@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg) * after we obtained our initial list of lockers, we will not wait for them. */ void -WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) +WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress) { List *holders = NIL; ListCell *lc; + int total = 0; + int done = 0; /* Done if no locks to wait for */ if (list_length(locktags) == 0) @@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) foreach(lc, locktags) { LOCKTAG *locktag = lfirst(lc); + int count; - holders = lappend(holders, GetLockConflicts(locktag, lockmode)); + holders = lappend(holders, + GetLockConflicts(locktag, lockmode, + progress ? &count : NULL)); + total += count; } + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total); + /* * Note: GetLockConflicts() never reports our own xid, hence we need not * check for that. Also, prepared xacts are not reported, which is fine @@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) while (VirtualTransactionIdIsValid(*lockholders)) { + /* + * If requested, publish who we're going to wait for. This is not + * 100% accurate if they're already gone, but we don't care. + */ + if (progress) + { + PGPROC *holder = BackendIdGetProc(lockholders->backendId); + + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); + } VirtualXactLock(*lockholders, true); lockholders++; + + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done); } } + if (progress) + { + const int index[] = { + PROGRESS_WAITFOR_TOTAL, + PROGRESS_WAITFOR_DONE, + PROGRESS_WAITFOR_CURRENT_PID + }; + const int64 values[] = { + 0, 0, 0 + }; + pgstat_progress_update_multi_param(3, index, values); + } list_free_deep(holders); } @@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) * Same as WaitForLockersMultiple, for a single lock tag. */ void -WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode) +WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress) { List *l; l = list_make1(&heaplocktag); - WaitForLockersMultiple(l, lockmode); + WaitForLockersMultiple(l, lockmode, progress); list_free(l); } diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c index 78fdbd6ff88..c8958766f1e 100644 --- a/src/backend/storage/lmgr/lock.c +++ b/src/backend/storage/lmgr/lock.c @@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * xacts merely awaiting such a lock are NOT reported. * * The result array is palloc'd and is terminated with an invalid VXID. + * *countp, if not null, is updated to the number of items set. * * Of course, the result could be out of date by the time it's returned, * so use of this function has to be thought about carefully. @@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * uses of the result. */ VirtualTransactionId * -GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) +GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp) { static VirtualTransactionId *vxids; LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid; @@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) LWLockRelease(partitionLock); vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } @@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c index 060ffe501ec..e81d6cc0562 100644 --- a/src/backend/utils/adt/amutils.c +++ b/src/backend/utils/adt/amutils.c @@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS) return indexam_property(fcinfo, propname, InvalidOid, relid, attno); } + +/* + * Return the name of the given phase, as used for progress reporting by the + * given AM. + */ +Datum +pg_indexam_progress_phasename(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + int32 phasenum = PG_GETARG_INT32(1); + IndexAmRoutine *routine; + char *name; + + routine = GetIndexAmRoutineByAmId(amoid, true); + if (routine == NULL || !routine->ambuildphasename) + PG_RETURN_NULL(); + + name = routine->ambuildphasename(phasenum); + if (!name) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(CStringGetTextDatum(name)); +} diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 90a817a25c5..7c2afe64272 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -470,6 +470,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS) cmdtype = PROGRESS_COMMAND_VACUUM; else if (pg_strcasecmp(cmd, "CLUSTER") == 0) cmdtype = PROGRESS_COMMAND_CLUSTER; + else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0) + cmdtype = PROGRESS_COMMAND_CREATE_INDEX; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index 653ddc976ba..09a7404267c 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +/* name of phase as used in progress reporting */ +typedef char *(*ambuildphasename_function) (int64 phasenum); + /* validate definition of an opclass for this AM */ typedef bool (*amvalidate_function) (Oid opclassoid); @@ -213,6 +216,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; diff --git a/src/include/access/genam.h b/src/include/access/genam.h index cad66513f62..e38e209e68c 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo { Relation index; /* the index being vacuumed */ bool analyze_only; /* ANALYZE (without any actual vacuum) */ + bool report_progress; /* emit progress.h status reports */ bool estimated_count; /* num_heap_tuples is an estimate */ int message_level; /* ereport level for progress messages */ double num_heap_tuples; /* tuples remaining in heap */ diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h index 473c6f29185..bc0f3027629 100644 --- a/src/include/access/nbtree.h +++ b/src/include/access/nbtree.h @@ -672,6 +672,16 @@ typedef BTScanOpaqueData *BTScanOpaque; #define SK_BT_DESC (INDOPTION_DESC << SK_BT_INDOPTION_SHIFT) #define SK_BT_NULLS_FIRST (INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT) +/* + * Constant definition for progress reporting. Phase numbers must match + * btbuildphasename. + */ +/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */ +#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN 2 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_1 3 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_2 4 +#define PROGRESS_BTREE_PHASE_LEAF_LOAD 5 + /* * external entry points for btree, in nbtree.c */ @@ -785,6 +795,7 @@ extern bytea *btoptions(Datum reloptions, bool validate); extern bool btproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +extern char *btbuildphasename(int64 phasenum); extern IndexTuple _bt_truncate(Relation rel, IndexTuple lastleft, IndexTuple firstright, BTScanInsert itup_key); extern int _bt_keep_natts_fast(Relation rel, IndexTuple lastleft, diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 29f7ed62379..f90e52382ba 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -115,6 +115,7 @@ extern double IndexBuildHeapScan(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, struct TableScanDescData *scan); @@ -123,6 +124,7 @@ extern double IndexBuildHeapRangeScan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber end_blockno, IndexBuildCallback callback, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index acf1131b521..22ce0d20247 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -924,6 +924,10 @@ proname => 'pg_index_column_has_property', provolatile => 's', prorettype => 'bool', proargtypes => 'regclass int4 text', prosrc => 'pg_index_column_has_property' }, +{ oid => '676', descr => 'return name of given index build phase', + proname => 'pg_indexam_progress_phasename', provolatile => 'i', + prorettype => 'text', proargtypes => 'oid int8', + prosrc => 'pg_indexam_progress_phasename' }, { oid => '339', proname => 'poly_same', prorettype => 'bool', @@ -5098,9 +5102,9 @@ proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't', provolatile => 's', proparallel => 'r', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', - proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}', + proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}', prosrc => 'pg_stat_get_progress_info' }, { oid => '3099', descr => 'statistics: information about currently active replication', diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h index 04542d9e923..45847f6dc35 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -57,4 +57,39 @@ #define PROGRESS_CLUSTER_COMMAND_CLUSTER 1 #define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL 2 +/* Progress parameters for CREATE INDEX */ +#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID 0 +#define PROGRESS_CREATEIDX_PHASE 1 /* AM-agnostic phase # */ +#define PROGRESS_CREATEIDX_SUBPHASE 2 /* phase # filled by AM */ +/* 3, 4 and 5 reserved for "waitfor" metrics */ +/* 6 and 7 reserved for "block number" metrics */ +#define PROGRESS_CREATEIDX_TUPLES_TOTAL 8 +#define PROGRESS_CREATEIDX_TUPLES_DONE 9 +#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL 10 +#define PROGRESS_CREATEIDX_PARTITIONS_DONE 11 + +/* Phases of CREATE INDEX */ +#define PROGRESS_CREATEIDX_PHASE_WAIT_1 1 +#define PROGRESS_CREATEIDX_PHASE_BUILD 2 +#define PROGRESS_CREATEIDX_PHASE_WAIT_2 3 +#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN 4 +#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN 5 +#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE 6 +#define PROGRESS_CREATEIDX_PHASE_WAIT_3 7 + +/* + * Subphases of CREATE INDEX, for index_build. + */ +#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE 1 +/* Additional phases are defined by each AM */ + +/* Lock holder wait counts */ +#define PROGRESS_WAITFOR_TOTAL 3 +#define PROGRESS_WAITFOR_DONE 4 +#define PROGRESS_WAITFOR_CURRENT_PID 5 + +/* Block numbers in a generic relation scan */ +#define PROGRESS_SCAN_BLOCKS_TOTAL 6 +#define PROGRESS_SCAN_BLOCKS_DONE 7 + #endif diff --git a/src/include/pgstat.h b/src/include/pgstat.h index c080fa6388f..6af9ab7e1d9 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -951,10 +951,11 @@ typedef enum ProgressCommandType { PROGRESS_COMMAND_INVALID, PROGRESS_COMMAND_VACUUM, - PROGRESS_COMMAND_CLUSTER + PROGRESS_COMMAND_CLUSTER, + PROGRESS_COMMAND_CREATE_INDEX } ProgressCommandType; -#define PGSTAT_NUM_PROGRESS_PARAM 10 +#define PGSTAT_NUM_PROGRESS_PARAM 12 /* ---------- * Shared-memory data structures diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h index 3d705faba5c..4f2872de35f 100644 --- a/src/include/storage/lmgr.h +++ b/src/include/storage/lmgr.h @@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel, extern bool ConditionalXactLockTableWait(TransactionId xid); /* Lock VXIDs, specified by conflicting locktags */ -extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode); -extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode); +extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress); +extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress); /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */ extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid); diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h index badf7fd682b..048947c50d4 100644 --- a/src/include/storage/lock.h +++ b/src/include/storage/lock.h @@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode); extern bool LockHasWaiters(const LOCKTAG *locktag, LOCKMODE lockmode, bool sessionLock); extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag, - LOCKMODE lockmode); + LOCKMODE lockmode, int *countp); extern void AtPrepare_Locks(void); extern void PostPrepare_Locks(TransactionId xid); extern int LockCheckConflicts(LockMethod lockMethodTable, diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index d5f309fbfbe..a9a2910fb55 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1856,7 +1856,33 @@ pg_stat_progress_cluster| SELECT s.pid, s.param6 AS heap_blks_total, s.param7 AS heap_blks_scanned, s.param8 AS index_rebuild_count - FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) + LEFT JOIN pg_database d ON ((s.datid = d.oid))); +pg_stat_progress_create_index| SELECT s.pid, + s.datid, + d.datname, + s.relid, + CASE s.param2 + WHEN 0 THEN 'initializing'::text + WHEN 1 THEN 'waiting for old snapshots'::text + WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param1)::oid, s.param3)), ''::text)) + WHEN 3 THEN 'waiting for writer snapshots'::text + WHEN 4 THEN 'index validation: scan index'::text + WHEN 5 THEN 'index validation: sort index scan results'::text + WHEN 6 THEN 'index validation: scan heap'::text + WHEN 7 THEN 'waiting for reader snapshots'::text + ELSE NULL::text + END AS phase, + s.param4 AS lockers_total, + s.param5 AS lockers_done, + s.param6 AS current_locker_pid, + s.param7 AS blocks_total, + s.param8 AS blocks_done, + s.param9 AS tuples_total, + s.param10 AS tuples_done, + s.param11 AS partitions_total, + s.param12 AS partitions_done + FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_progress_vacuum| SELECT s.pid, s.datid, @@ -1878,7 +1904,7 @@ pg_stat_progress_vacuum| SELECT s.pid, s.param5 AS index_vacuum_count, s.param6 AS max_dead_tuples, s.param7 AS num_dead_tuples - FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_replication| SELECT s.pid, s.usesysid, -- 2.17.1 --sm4nu43k4a2Rpi4c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0003-report-progress-of-hash-indexes.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v5 1/2] Report progress of CREATE INDEX operations @ 2019-01-02 19:14 Alvaro Herrera <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Alvaro Herrera @ 2019-01-02 19:14 UTC (permalink / raw) --- contrib/amcheck/verify_nbtree.c | 2 +- contrib/bloom/blinsert.c | 2 +- contrib/bloom/blutils.c | 1 + doc/src/sgml/indexam.sgml | 13 ++ doc/src/sgml/monitoring.sgml | 227 +++++++++++++++++++++++++- src/backend/access/brin/brin.c | 5 +- src/backend/access/gin/gininsert.c | 2 +- src/backend/access/gin/ginutil.c | 1 + src/backend/access/gist/gist.c | 1 + src/backend/access/gist/gistbuild.c | 2 +- src/backend/access/hash/hash.c | 3 +- src/backend/access/nbtree/nbtree.c | 9 + src/backend/access/nbtree/nbtsort.c | 57 ++++++- src/backend/access/nbtree/nbtutils.c | 24 +++ src/backend/access/spgist/spginsert.c | 2 +- src/backend/access/spgist/spgutils.c | 1 + src/backend/catalog/index.c | 147 ++++++++++++++++- src/backend/catalog/system_views.sql | 27 +++ src/backend/commands/indexcmds.c | 52 +++++- src/backend/storage/ipc/standby.c | 2 +- src/backend/storage/lmgr/lmgr.c | 46 +++++- src/backend/storage/lmgr/lock.c | 7 +- src/backend/utils/adt/amutils.c | 23 +++ src/backend/utils/adt/pgstatfuncs.c | 2 + src/include/access/amapi.h | 4 + src/include/access/genam.h | 1 + src/include/access/nbtree.h | 11 ++ src/include/catalog/index.h | 2 + src/include/catalog/pg_proc.dat | 10 +- src/include/commands/progress.h | 36 ++++ src/include/pgstat.h | 5 +- src/include/storage/lmgr.h | 4 +- src/include/storage/lock.h | 2 +- 33 files changed, 695 insertions(+), 38 deletions(-) diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c index 964200a7678..99f6ed6bc44 100644 --- a/contrib/amcheck/verify_nbtree.c +++ b/contrib/amcheck/verify_nbtree.c @@ -534,7 +534,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool readonly, RelationGetRelationName(state->rel), RelationGetRelationName(state->heaprel)); - IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, + IndexBuildHeapScan(state->heaprel, state->rel, indexinfo, true, false, bt_tuple_present_callback, (void *) state, scan); ereport(DEBUG1, diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c index e43fbe0005f..947ee74881f 100644 --- a/contrib/bloom/blinsert.c +++ b/contrib/bloom/blinsert.c @@ -141,7 +141,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo) initCachedPage(&buildstate); /* Do the heap scan */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, bloomBuildCallback, (void *) &buildstate, NULL); diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c index 64583765787..697a37a384b 100644 --- a/contrib/bloom/blutils.c +++ b/contrib/bloom/blutils.c @@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = blcostestimate; amroutine->amoptions = bloptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = blvalidate; amroutine->ambeginscan = blbeginscan; amroutine->amrescan = blrescan; diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml index 05102724ead..fa4a3d0d131 100644 --- a/doc/src/sgml/indexam.sgml +++ b/doc/src/sgml/indexam.sgml @@ -127,6 +127,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; @@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno, <para> <programlisting> +char * +ambuildphasename (int64 phasenum); +</programlisting> + Return the textual name of the given build phase number. + The phase numbers are those reported during an index build via the + <function>pgstat_progress_update_param</function> interface. + The phase names are then exposed in the + <structname>pg_stat_progress_create_index</structname> view. + </para> + + <para> +<programlisting> bool amvalidate (Oid opclassoid); </programlisting> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 0e73cdcddab..47708570f84 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -336,6 +336,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </entry> </row> + <row> + <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry> + <entry>One row for each backend running <command>CREATE INDEX</command>, showing + current progress. + See <xref linkend='create-index-progress-reporting'/>. + </entry> + </row> + <row> <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry> <entry>One row for each backend (including autovacuum worker processes) running @@ -3376,11 +3384,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <para> <productname>PostgreSQL</productname> has the ability to report the progress of - certain commands during command execution. Currently, the only command - which supports progress reporting is <command>VACUUM</command>. This may be - expanded in the future. + certain commands during command execution. Currently, the only commands + which support progress reporting are <command>CREATE INDEX</command> and + <command>VACUUM</command>. This may be expanded in the future. </para> + <sect2 id="create-index-progress-reporting"> + <title>CREATE INDEX Progress Reporting</title> + + <para> + Whenever <command>CREATE INDEX</command> is running, the + <structname>pg_stat_progress_create_index</structname> view will contain + one row for each backend that is currently creating indexes. The tables + below describe the information that will be reported and provide information + about how to interpret it. + </para> + + <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index"> + <title><structname>pg_stat_progress_create_index</structname> View</title> + <tgroup cols="3"> + <thead> + <row> + <entry>Column</entry> + <entry>Type</entry> + <entry>Description</entry> + </row> + </thead> + + <tbody> + <row> + <entry><structfield>pid</structfield></entry> + <entry><type>integer</type></entry> + <entry>Process ID of backend.</entry> + </row> + <row> + <entry><structfield>datid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>datname</structfield></entry> + <entry><type>name</type></entry> + <entry>Name of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>relid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the table on which the index is being created.</entry> + </row> + <row> + <entry><structfield>phase</structfield></entry> + <entry><type>text</type></entry> + <entry> + Current processing phase of index creation. See <xref linkend='create-index-phases'/>. + </entry> + </row> + <row> + <entry><structfield>lockers_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of lockers to wait for, when applicable. + </entry> + </row> + <row> + <entry><structfield>lockers_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of lockers already waited for. + </entry> + </row> + <row> + <entry><structfield>current_locked_pid</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Process ID of the locker currently being waited for. + </entry> + </row> + <row> + <entry><structfield>blocks_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of blocks to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>blocks_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of blocks already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of tuples to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of tuples already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>partitions_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + total number of partitions on which the index is to be created. + </entry> + </row> + <row> + <entry><structfield>partitions_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + number of partitions on which the index has been completed. + </entry> + </row> + </tbody> + </tgroup> + </table> + + <table id="create-index-phases"> + <title>CREATE INDEX phases</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Phase</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry><literal>initializing</literal></entry> + <entry> + <command>CREATE INDEX</command> is preparing to create the index. This + phase is expected to be very brief. + </entry> + </row> + <row> + <entry><literal>waiting for old snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>building index</literal></entry> + <entry> + The index is being built by the access method-specific code. In this phase, + access methods that support progress reporting fill in their own progress data, + and the subphase is indicated in this column. Typically, + <structname>blocks_total</structname> and <structname>blocks_done</structname> + will contain progress data, as well as potentially + <structname>tuples_total</structname> and <structname>tuples_done</structname>. + </entry> + </row> + <row> + <entry><literal>waiting for writer snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially write into the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>validating index scan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching + for tuples that need to be validated. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the index) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>sorting index scan results</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the + previous phase. + </entry> + </row> + <row> + <entry><literal>validating index heapscan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the table + to validate the index tuples collected in the previous two phases. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the table) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>waiting for reader snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. This + phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + </tbody> + </tgroup> + </table> + + </sect2> + <sect2 id="vacuum-progress-reporting"> <title>VACUUM Progress Reporting</title> diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index 8f008dd0080..5d1aff34080 100644 --- a/src/backend/access/brin/brin.c +++ b/src/backend/access/brin/brin.c @@ -111,6 +111,7 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = brincostestimate; amroutine->amoptions = brinoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = brinvalidate; amroutine->ambeginscan = brinbeginscan; amroutine->amrescan = brinrescan; @@ -718,7 +719,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Now scan the relation. No syncscan allowed here because we want the * heap blocks in physical order. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false, brinbuildCallback, (void *) state, NULL); /* process the final batch */ @@ -1234,7 +1235,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel, * by transactions that are still in progress, among other corner cases. */ state->bs_currRangeStart = heapBlk; - IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, + IndexBuildHeapRangeScan(heapRel, state->bs_irel, indexInfo, false, true, false, heapBlk, scanNumBlks, brinbuildCallback, (void *) state, NULL); diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c index 524ac5be8b5..838de4c1ec3 100644 --- a/src/backend/access/gin/gininsert.c +++ b/src/backend/access/gin/gininsert.c @@ -394,7 +394,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Do the heap scan. We disallow sync scan here because dataPlaceToPage * prefers to receive tuples in TID order. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, false, false, ginBuildCallback, (void *) &buildstate, NULL); /* dump remaining entries to the index */ diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c index afc20232ace..d2360eeafb0 100644 --- a/src/backend/access/gin/ginutil.c +++ b/src/backend/access/gin/ginutil.c @@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gincostestimate; amroutine->amoptions = ginoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = ginvalidate; amroutine->ambeginscan = ginbeginscan; amroutine->amrescan = ginrescan; diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index b75b3a8dacd..0dc36af1e0c 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gistcostestimate; amroutine->amoptions = gistoptions; amroutine->amproperty = gistproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = gistvalidate; amroutine->ambeginscan = gistbeginscan; amroutine->amrescan = gistrescan; diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index bd142a3560d..015f874cc93 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -204,7 +204,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) /* * Do the heap scan. */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, gistBuildCallback, (void *) &buildstate, NULL); /* diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c index f1f01a0956d..fc7db5d6a13 100644 --- a/src/backend/access/hash/hash.c +++ b/src/backend/access/hash/hash.c @@ -82,6 +82,7 @@ hashhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = hashcostestimate; amroutine->amoptions = hashoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = hashvalidate; amroutine->ambeginscan = hashbeginscan; amroutine->amrescan = hashrescan; @@ -159,7 +160,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo) buildstate.heapRel = heap; /* do the heap scan */ - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, hashbuildCallback, (void *) &buildstate, NULL); if (buildstate.spool) diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index 98917de2efd..f9d32c958f6 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -22,6 +22,7 @@ #include "access/nbtxlog.h" #include "access/relscan.h" #include "access/xlog.h" +#include "commands/progress.h" #include "commands/vacuum.h" #include "miscadmin.h" #include "nodes/execnodes.h" @@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = btcostestimate; amroutine->amoptions = btoptions; amroutine->amproperty = btproperty; + amroutine->ambuildphasename = btbuildphasename; amroutine->amvalidate = btvalidate; amroutine->ambeginscan = btbeginscan; amroutine->amrescan = btrescan; @@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (needLock) UnlockRelationForExtension(rel, ExclusiveLock); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + num_pages); + /* Quit if we've scanned the whole relation */ if (blkno >= num_pages) break; @@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, for (; blkno < num_pages; blkno++) { btvacuumpage(&vstate, blkno, blkno); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blkno); } } diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index dc398e11867..309a2082063 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -65,6 +65,7 @@ #include "access/xlog.h" #include "access/xloginsert.h" #include "catalog/index.h" +#include "commands/progress.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/smgr.h" @@ -288,7 +289,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate, static void _bt_leader_participate_as_worker(BTBuildState *buildstate); static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem); + Sharedsort *sharedsort2, int sortmem, + bool progress); /* @@ -384,6 +386,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Save as primary spool */ buildstate->spool = btspool; + /* Report heap scan phase started */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN); + /* Attempt to launch parallel worker scan when required */ if (indexInfo->ii_ParallelWorkers > 0) _bt_begin_parallel(buildstate, indexInfo->ii_Concurrent, @@ -470,13 +476,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Fill spool using either serial or parallel heap scan */ if (!buildstate->btleader) - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, true, _bt_build_callback, (void *) buildstate, NULL); else reltuples = _bt_parallel_heapscan(buildstate, &indexInfo->ii_BrokenHotChain); + /* + * Set the progress target for the next phase. Reset the block number + * values set by IndexBuildHeapScan + */ + { + const int index[] = { + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE + }; + const int64 val[] = { + buildstate->indtuples, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } + /* okay, all heap tuples are spooled */ if (buildstate->spool2 && !buildstate->havedead) { @@ -525,9 +549,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) } #endif /* BTREE_BUILD_STATS */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_1); tuplesort_performsort(btspool->sortstate); if (btspool2) + { + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_2); tuplesort_performsort(btspool2->sortstate); + } wstate.heap = btspool->heap; wstate.index = btspool->index; @@ -543,6 +573,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) wstate.btws_pages_written = 0; wstate.btws_zeropage = NULL; /* until needed */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_LEAF_LOAD); _bt_load(&wstate, btspool, btspool2); } @@ -1078,6 +1110,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index); ScanKey indexScanKey = NULL; SortSupport sortKeys; + long tuples_done = 0L; if (merge) { @@ -1170,6 +1203,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) _bt_buildadd(wstate, state, itup2); itup2 = tuplesort_getindextuple(btspool2->sortstate, true); } + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } pfree(sortKeys); } @@ -1184,6 +1221,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) state = _bt_pagestate(wstate, 0); _bt_buildadd(wstate, state, itup); + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } } @@ -1318,6 +1359,10 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request) btshared->brokenhotchain = false; heap_parallelscan_initialize(&btshared->heapdesc, btspool->heap, snapshot); + /* Report total number of blocks to scan */ + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + btshared->heapdesc.phs_nblocks); + /* * Store shared tuplesort-private state, for which we reserved space. * Then, initialize opaque state using tuplesort routine. @@ -1493,7 +1538,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate) /* Perform work common to all participants */ _bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared, btleader->sharedsort, btleader->sharedsort2, - sortmem); + sortmem, true); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1584,7 +1629,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) /* Perform sorting of spool, and possibly a spool2 */ sortmem = maintenance_work_mem / btshared->scantuplesortstates; _bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort, - sharedsort2, sortmem); + sharedsort2, sortmem, false); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1613,7 +1658,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem) + Sharedsort *sharedsort2, int sortmem, bool progress) { SortCoordinate coordinate; BTBuildState buildstate; @@ -1672,7 +1717,7 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, indexInfo->ii_Concurrent = btshared->isconcurrent; scan = heap_beginscan_parallel(btspool->heap, &btshared->heapdesc); reltuples = IndexBuildHeapScan(btspool->heap, btspool->index, indexInfo, - true, _bt_build_callback, + true, progress, _bt_build_callback, (void *) &buildstate, scan); /* diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c index 2c05fb5e451..295c22b2e5b 100644 --- a/src/backend/access/nbtree/nbtutils.c +++ b/src/backend/access/nbtree/nbtutils.c @@ -20,6 +20,7 @@ #include "access/nbtree.h" #include "access/reloptions.h" #include "access/relscan.h" +#include "commands/progress.h" #include "miscadmin.h" #include "utils/array.h" #include "utils/lsyscache.h" @@ -2082,6 +2083,29 @@ btproperty(Oid index_oid, int attno, } } +/* + * btbuildphasename() -- Return name of index build phase. + */ +char * +btbuildphasename(int64 phasenum) +{ + switch (phasenum) + { + case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE: + return "initializing (1 of 5)"; + case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN: + return "table scan (2 of 5)"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_1: + return "sorting tuples, spool 1 (3 of 5)"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_2: + return "sorting tuples, spool 2 (4 of 5)"; + case PROGRESS_BTREE_PHASE_LEAF_LOAD: + return "btree tuple loading (5 of 5)"; + default: + return NULL; + } +} + /* * _bt_nonkey_truncate() -- create tuple without non-key suffix attributes. * diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c index f428a151385..1bc671c7238 100644 --- a/src/backend/access/spgist/spginsert.c +++ b/src/backend/access/spgist/spginsert.c @@ -142,7 +142,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo) "SP-GiST build temporary context", ALLOCSET_DEFAULT_SIZES); - reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, + reltuples = IndexBuildHeapScan(heap, index, indexInfo, true, false, spgistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c index 8e63c1fad25..45472db147b 100644 --- a/src/backend/access/spgist/spgutils.c +++ b/src/backend/access/spgist/spgutils.c @@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = spgcostestimate; amroutine->amoptions = spgoptions; amroutine->amproperty = spgproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = spgvalidate; amroutine->ambeginscan = spgbeginscan; amroutine->amrescan = spgrescan; diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index d16c3d0ea50..110f13d1e46 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -49,8 +49,9 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" -#include "commands/tablecmds.h" #include "commands/event_trigger.h" +#include "commands/progress.h" +#include "commands/tablecmds.h" #include "commands/trigger.h" #include "executor/executor.h" #include "miscadmin.h" @@ -58,6 +59,7 @@ #include "nodes/nodeFuncs.h" #include "optimizer/optimizer.h" #include "parser/parser.h" +#include "pgstat.h" #include "rewrite/rewriteManip.h" #include "storage/bufmgr.h" #include "storage/lmgr.h" @@ -1597,7 +1599,7 @@ index_drop(Oid indexId, bool concurrent) * to acquire an exclusive lock on our table. The lock code will * detect deadlock and error out properly. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * No more predicate locks will be acquired on this index, and we're @@ -1641,7 +1643,7 @@ index_drop(Oid indexId, bool concurrent) * Wait till every transaction that saw the old index state has * finished. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * Re-open relations to allow us to complete our actions. @@ -2291,6 +2293,25 @@ index_build(Relation heapRelation, save_sec_context | SECURITY_RESTRICTED_OPERATION); save_nestlevel = NewGUCNestLevel(); + /* Set up initial progress report status */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_BUILD, + PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE, + 0, 0, 0, 0 + }; + + pgstat_progress_update_multi_param(6, index, val); + } + /* * Call the access method's build procedure */ @@ -2417,6 +2438,12 @@ index_build(Relation heapRelation, * do so here because the AM might reject some of the tuples for its own * reasons, such as being unable to store NULLs. * + * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go + * along. Also, if that flag is true and a scan descriptor is not passed (ie. + * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is + * updated at the beginning. For parallel scans, caller is expected to have + * set the total number of blocks prior to calling this function. + * * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect * any potentially broken HOT chains. Currently, we set this if there are * any RECENTLY_DEAD or DELETE_IN_PROGRESS entries in a HOT chain, without @@ -2428,13 +2455,14 @@ IndexBuildHeapScan(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, HeapScanDesc scan) { return IndexBuildHeapRangeScan(heapRelation, indexRelation, indexInfo, allow_sync, - false, + false, progress, 0, InvalidBlockNumber, callback, callback_state, scan); } @@ -2445,6 +2473,13 @@ IndexBuildHeapScan(Relation heapRelation, * passing InvalidBlockNumber as numblocks. Note that restricting the range * to scan cannot be done when requesting syncscan. * + * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go + * along. Also, if that flag is true and a scan descriptor is not passed (ie. + * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is + * updated at the beginning. For parallel scans, caller is expected to have + * set the total number of blocks prior to calling this function. Note this + * only works for full-relation scans. + * * When "anyvisible" mode is requested, all tuples visible to any transaction * are indexed and counted as live, including those inserted or deleted by * transactions that are still in progress. @@ -2455,6 +2490,7 @@ IndexBuildHeapRangeScan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -2476,6 +2512,8 @@ IndexBuildHeapRangeScan(Relation heapRelation, TransactionId OldestXmin; BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; + BlockNumber blocks_done = 0; + BlockNumber previous_blkno = InvalidBlockNumber; /* * sanity checks @@ -2545,6 +2583,13 @@ IndexBuildHeapRangeScan(Relation heapRelation, NULL, /* scan key */ true, /* buffer access strategy OK */ allow_sync); /* syncscan OK? */ + + if (progress) + { + Assert(scan->rs_numblocks == InvalidBlockNumber); + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + scan->rs_nblocks); + } } else { @@ -2592,6 +2637,46 @@ IndexBuildHeapRangeScan(Relation heapRelation, CHECK_FOR_INTERRUPTS(); + /* Report scan progress, if asked to. */ + if (progress && + ((previous_blkno == InvalidBlockNumber) || + (scan->rs_cblock != previous_blkno))) + { + /* we only do progress for full table scans */ + Assert(numblocks == InvalidBlockNumber); + + /* + * Report the number of blocks we've moved forward. + * + * Parallel workers cause the leader process to skip some blocks, + * so we subtract the current block number to the previous one to + * determine how many have been read in total; but be careful when + * we wrap around the last block in the table. + */ + if (scan->rs_cblock > previous_blkno) + blocks_done += scan->rs_cblock - previous_blkno; + else if (previous_blkno == InvalidBlockNumber) + { + /* + * How many blocks have been read since the scan started. + * Should normally be zero. + */ + blocks_done += scan->rs_cblock - + (scan->rs_parallel ? scan->rs_parallel->phs_startblock : + scan->rs_startblock); + Assert(blocks_done == 0); + } + else + { + /* wrapped around */ + blocks_done += scan->rs_nblocks - previous_blkno + scan->rs_cblock; + } + + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blocks_done); + previous_blkno = scan->rs_cblock; + } + /* * When dealing with a HOT-chain of updated tuples, we want to index * the values of the live tuple (if any), but index it under the TID @@ -3133,6 +3218,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) int save_sec_context; int save_nestlevel; + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN, + 0, 0, 0, 0 + }; + pgstat_progress_update_multi_param(5, index, val); + } + /* Open and lock the parent heap relation */ heapRelation = table_open(heapId, ShareUpdateExclusiveLock); /* And the target index relation */ @@ -3163,6 +3263,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) */ ivinfo.index = indexRelation; ivinfo.analyze_only = false; + ivinfo.report_progress = true; /* XXX only for btree? */ ivinfo.estimated_count = true; ivinfo.message_level = DEBUG2; ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples; @@ -3180,15 +3281,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) NULL, false); state.htups = state.itups = state.tups_inserted = 0; + /* ambulkdelete updates progress metrics */ (void) index_bulk_delete(&ivinfo, NULL, validate_index_callback, (void *) &state); /* Execute the sort */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } tuplesort_performsort(state.tuplesort); /* - * Now scan the heap and "merge" it with the index + * Now scan the heap and "merge" it with the index. */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE + }; + + pgstat_progress_update_multi_param(1, index, val); + } validate_index_heapscan(heapRelation, indexRelation, indexInfo, @@ -3291,6 +3416,7 @@ validate_index_heapscan(Relation heapRelation, BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; bool in_index[MaxHeapTuplesPerPage]; + BlockNumber previous_blkno = InvalidBlockNumber; /* state variables for the merge */ ItemPointer indexcursor = NULL; @@ -3330,6 +3456,9 @@ validate_index_heapscan(Relation heapRelation, true, /* buffer access strategy OK */ false); /* syncscan not OK */ + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + scan->rs_nblocks); + /* * Scan all tuples matching the snapshot. */ @@ -3343,6 +3472,14 @@ validate_index_heapscan(Relation heapRelation, state->htups += 1; + if ((previous_blkno == InvalidBlockNumber) || + (scan->rs_cblock != previous_blkno)) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + scan->rs_cblock); + previous_blkno = scan->rs_cblock; + } + /* * As commented in IndexBuildHeapScan, we should index heap-only * tuples under the TIDs of their root tuples; so when we advance onto diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 3e229c693c4..3cbe6f8af2a 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -906,6 +906,33 @@ CREATE VIEW pg_stat_progress_vacuum AS FROM pg_stat_get_progress_info('VACUUM') AS S LEFT JOIN pg_database D ON S.datid = D.oid; +CREATE VIEW pg_stat_progress_create_index AS + SELECT + S.pid AS pid, S.datid AS datid, D.datname AS datname, + S.relid AS relid, + CASE S.param2 WHEN 0 THEN 'initializing (phase 1 of 8)' + WHEN 1 THEN 'waiting for old snapshots (phase 2 of 8)' + WHEN 2 THEN 'building index (3 of 8)' || + COALESCE((': ' || pg_indexam_progress_phasename(S.param1::oid, S.param3)), + '') + WHEN 3 THEN 'waiting for writer snapshots (phase 4 of 8)' + WHEN 4 THEN 'index validation: scan index (phase 5 of 8)' + WHEN 5 THEN 'index validation: sort index scan results (phase 6 of 8)' + WHEN 6 THEN 'index validation: scan heap (phase 7 of 8)' + WHEN 7 THEN 'waiting for reader snapshots (phase 8 of 8)' + END as phase, + S.param4 AS lockers_total, + S.param5 AS lockers_done, + S.param6 AS current_locker_pid, + S.param7 AS blocks_total, + S.param8 AS blocks_done, + S.param9 AS tuples_total, + S.param10 AS tuples_done, + S.param11 AS partitions_total, + S.param12 AS partitions_done + FROM pg_stat_get_progress_info('CREATE INDEX') AS S + LEFT JOIN pg_database D ON S.datid = D.oid; + CREATE VIEW pg_user_mappings AS SELECT U.oid AS umid, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 5dcedc337aa..8f8587ac1e3 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -35,6 +35,7 @@ #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" +#include "commands/progress.h" #include "commands/tablecmds.h" #include "commands/tablespace.h" #include "mb/pg_wchar.h" @@ -46,10 +47,12 @@ #include "parser/parse_func.h" #include "parser/parse_oper.h" #include "partitioning/partdesc.h" +#include "pgstat.h" #include "rewrite/rewriteManip.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/acl.h" #include "utils/builtins.h" #include "utils/fmgroids.h" @@ -368,6 +371,15 @@ DefineIndex(Oid relationId, Snapshot snapshot; int i; + + /* + * Start progress report. If we're building a partition, this was already + * done. + */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX, + relationId); + /* * count key attributes in index */ @@ -584,6 +596,9 @@ DefineIndex(Oid relationId, accessMethodId = accessMethodForm->oid; amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler); + pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID, + accessMethodId); + if (stmt->unique && !amRoutine->amcanunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -864,6 +879,11 @@ DefineIndex(Oid relationId, if (!OidIsValid(indexRelationId)) { table_close(rel, NoLock); + + /* If this is the top-level index, we're done */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -889,6 +909,9 @@ DefineIndex(Oid relationId, TupleDesc parentDesc; Oid *opfamOids; + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL, + nparts); + memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts); parentDesc = CreateTupleDescCopy(RelationGetDescr(rel)); @@ -1039,6 +1062,8 @@ DefineIndex(Oid relationId, skip_build, quiet); } + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, + i + 1); pfree(attmap); } @@ -1073,6 +1098,8 @@ DefineIndex(Oid relationId, * Indexes on partitioned tables are not themselves built, so we're * done here. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); return address; } @@ -1080,6 +1107,11 @@ DefineIndex(Oid relationId, { /* Close the heap and we're done, in the non-concurrent case */ table_close(rel, NoLock); + + /* If this is the top-level index, we're done. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -1131,7 +1163,9 @@ DefineIndex(Oid relationId, * exclusive lock on our table. The lock code will detect deadlock and * error out properly. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_1); + WaitForLockers(heaplocktag, ShareLock, true); /* * At this moment we are sure that there are no transactions with the @@ -1195,7 +1229,9 @@ DefineIndex(Oid relationId, * We once again wait until no transaction can have the table open with * the index marked as read-only for updates. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_2); + WaitForLockers(heaplocktag, ShareLock, true); /* * Now take the "reference snapshot" that will be used by validate_index() @@ -1281,6 +1317,9 @@ DefineIndex(Oid relationId, old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false, PROC_IS_AUTOVACUUM | PROC_IN_VACUUM, &n_old_snapshots); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_3); + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots); for (i = 0; i < n_old_snapshots; i++) { @@ -1316,7 +1355,14 @@ DefineIndex(Oid relationId, } if (VirtualTransactionIdIsValid(old_snapshots[i])) + { + PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId); + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); VirtualXactLock(old_snapshots[i], true); + } + + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1); } /* @@ -1339,6 +1385,8 @@ DefineIndex(Oid relationId, */ UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock); + pgstat_progress_end_command(); + return address; } diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index 4d10e57a803..a0a2b964703 100644 --- a/src/backend/storage/ipc/standby.c +++ b/src/backend/storage/ipc/standby.c @@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag) */ VirtualTransactionId *backends; - backends = GetLockConflicts(&locktag, AccessExclusiveLock); + backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL); ResolveRecoveryConflictWithVirtualXIDs(backends, PROCSIG_RECOVERY_CONFLICT_LOCK); } diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c index e688ba81170..0b04b093782 100644 --- a/src/backend/storage/lmgr/lmgr.c +++ b/src/backend/storage/lmgr/lmgr.c @@ -19,9 +19,12 @@ #include "access/transam.h" #include "access/xact.h" #include "catalog/catalog.h" +#include "commands/progress.h" #include "miscadmin.h" +#include "pgstat.h" #include "storage/lmgr.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/inval.h" @@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg) * after we obtained our initial list of lockers, we will not wait for them. */ void -WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) +WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress) { List *holders = NIL; ListCell *lc; + int total = 0; + int done = 0; /* Done if no locks to wait for */ if (list_length(locktags) == 0) @@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) foreach(lc, locktags) { LOCKTAG *locktag = lfirst(lc); + int count; - holders = lappend(holders, GetLockConflicts(locktag, lockmode)); + holders = lappend(holders, + GetLockConflicts(locktag, lockmode, + progress ? &count : NULL)); + total += count; } + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total); + /* * Note: GetLockConflicts() never reports our own xid, hence we need not * check for that. Also, prepared xacts are not reported, which is fine @@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) while (VirtualTransactionIdIsValid(*lockholders)) { + /* + * If requested, publish who we're going to wait for. This is not + * 100% accurate if they're already gone, but we don't care. + */ + if (progress) + { + PGPROC *holder = BackendIdGetProc(lockholders->backendId); + + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); + } VirtualXactLock(*lockholders, true); lockholders++; + + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done); } } + if (progress) + { + const int index[] = { + PROGRESS_WAITFOR_TOTAL, + PROGRESS_WAITFOR_DONE, + PROGRESS_WAITFOR_CURRENT_PID + }; + const int64 values[] = { + 0, 0, 0 + }; + pgstat_progress_update_multi_param(3, index, values); + } list_free_deep(holders); } @@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) * Same as WaitForLockersMultiple, for a single lock tag. */ void -WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode) +WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress) { List *l; l = list_make1(&heaplocktag); - WaitForLockersMultiple(l, lockmode); + WaitForLockersMultiple(l, lockmode, progress); list_free(l); } diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c index 78fdbd6ff88..c8958766f1e 100644 --- a/src/backend/storage/lmgr/lock.c +++ b/src/backend/storage/lmgr/lock.c @@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * xacts merely awaiting such a lock are NOT reported. * * The result array is palloc'd and is terminated with an invalid VXID. + * *countp, if not null, is updated to the number of items set. * * Of course, the result could be out of date by the time it's returned, * so use of this function has to be thought about carefully. @@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * uses of the result. */ VirtualTransactionId * -GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) +GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp) { static VirtualTransactionId *vxids; LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid; @@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) LWLockRelease(partitionLock); vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } @@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c index 060ffe501ec..e81d6cc0562 100644 --- a/src/backend/utils/adt/amutils.c +++ b/src/backend/utils/adt/amutils.c @@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS) return indexam_property(fcinfo, propname, InvalidOid, relid, attno); } + +/* + * Return the name of the given phase, as used for progress reporting by the + * given AM. + */ +Datum +pg_indexam_progress_phasename(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + int32 phasenum = PG_GETARG_INT32(1); + IndexAmRoutine *routine; + char *name; + + routine = GetIndexAmRoutineByAmId(amoid, true); + if (routine == NULL || !routine->ambuildphasename) + PG_RETURN_NULL(); + + name = routine->ambuildphasename(phasenum); + if (!name) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(CStringGetTextDatum(name)); +} diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 69f72657792..05f9da6a4ad 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS) /* Translate command name into command type code. */ if (pg_strcasecmp(cmd, "VACUUM") == 0) cmdtype = PROGRESS_COMMAND_VACUUM; + else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0) + cmdtype = PROGRESS_COMMAND_CREATE_INDEX; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index 653ddc976ba..09a7404267c 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +/* name of phase as used in progress reporting */ +typedef char *(*ambuildphasename_function) (int64 phasenum); + /* validate definition of an opclass for this AM */ typedef bool (*amvalidate_function) (Oid opclassoid); @@ -213,6 +216,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; diff --git a/src/include/access/genam.h b/src/include/access/genam.h index c4aba39496f..f77d9eea8e8 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo { Relation index; /* the index being vacuumed */ bool analyze_only; /* ANALYZE (without any actual vacuum) */ + bool report_progress; /* emit progress.h status reports */ bool estimated_count; /* num_heap_tuples is an estimate */ int message_level; /* ereport level for progress messages */ double num_heap_tuples; /* tuples remaining in heap */ diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h index 60622ea7906..223aa138837 100644 --- a/src/include/access/nbtree.h +++ b/src/include/access/nbtree.h @@ -488,6 +488,16 @@ typedef BTScanOpaqueData *BTScanOpaque; #define SK_BT_DESC (INDOPTION_DESC << SK_BT_INDOPTION_SHIFT) #define SK_BT_NULLS_FIRST (INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT) +/* + * Constant definition for progress reporting. Phase numbers must match + * btbuildphasename. + */ +/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */ +#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN 2 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_1 3 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_2 4 +#define PROGRESS_BTREE_PHASE_LEAF_LOAD 5 + /* * external entry points for btree, in nbtree.c */ @@ -600,6 +610,7 @@ extern bytea *btoptions(Datum reloptions, bool validate); extern bool btproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +extern char *btbuildphasename(int64 phasenum); extern IndexTuple _bt_nonkey_truncate(Relation rel, IndexTuple itup); extern bool _bt_check_natts(Relation rel, Page page, OffsetNumber offnum); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 330c481a8b7..fb713d50f96 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -114,6 +114,7 @@ extern double IndexBuildHeapScan(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, struct HeapScanDescData *scan); @@ -122,6 +123,7 @@ extern double IndexBuildHeapRangeScan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber end_blockno, IndexBuildCallback callback, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 5bb56b2c639..0250f8533a9 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -917,6 +917,10 @@ proname => 'pg_index_column_has_property', provolatile => 's', prorettype => 'bool', proargtypes => 'regclass int4 text', prosrc => 'pg_index_column_has_property' }, +{ oid => '676', descr => 'return name of given index build phase', + proname => 'pg_indexam_progress_phasename', provolatile => 'i', + prorettype => 'text', proargtypes => 'oid int8', + prosrc => 'pg_indexam_progress_phasename' }, { oid => '339', proname => 'poly_same', prorettype => 'bool', @@ -5079,9 +5083,9 @@ proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't', provolatile => 's', proparallel => 'r', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', - proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}', + proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}', prosrc => 'pg_stat_get_progress_info' }, { oid => '3099', descr => 'statistics: information about currently active replication', diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h index 9858b36a383..56bd6125594 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -34,4 +34,40 @@ #define PROGRESS_VACUUM_PHASE_TRUNCATE 5 #define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6 + +/* Progress parameters for CREATE INDEX */ +#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID 0 +#define PROGRESS_CREATEIDX_PHASE 1 /* AM-agnostic phase # */ +#define PROGRESS_CREATEIDX_SUBPHASE 2 /* phase # filled by AM */ +/* 3, 4 and 5 reserved for "waitfor" metrics */ +/* 6 and 7 reserved for "block number" metrics */ +#define PROGRESS_CREATEIDX_TUPLES_TOTAL 8 +#define PROGRESS_CREATEIDX_TUPLES_DONE 9 +#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL 10 +#define PROGRESS_CREATEIDX_PARTITIONS_DONE 11 + +/* Phases of CREATE INDEX */ +#define PROGRESS_CREATEIDX_PHASE_WAIT_1 1 +#define PROGRESS_CREATEIDX_PHASE_BUILD 2 +#define PROGRESS_CREATEIDX_PHASE_WAIT_2 3 +#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN 4 +#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN 5 +#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE 6 +#define PROGRESS_CREATEIDX_PHASE_WAIT_3 7 + +/* + * Subphases of CREATE INDEX, for index_build. + */ +#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE 1 +/* Additional phases are defined by each AM */ + +/* Lock holder wait counts */ +#define PROGRESS_WAITFOR_TOTAL 3 +#define PROGRESS_WAITFOR_DONE 4 +#define PROGRESS_WAITFOR_CURRENT_PID 5 + +/* Block numbers in a generic relation scan */ +#define PROGRESS_SCAN_BLOCKS_TOTAL 6 +#define PROGRESS_SCAN_BLOCKS_DONE 7 + #endif diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 88a75fb798e..f931ead1c27 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -934,10 +934,11 @@ typedef enum typedef enum ProgressCommandType { PROGRESS_COMMAND_INVALID, - PROGRESS_COMMAND_VACUUM + PROGRESS_COMMAND_VACUUM, + PROGRESS_COMMAND_CREATE_INDEX } ProgressCommandType; -#define PGSTAT_NUM_PROGRESS_PARAM 10 +#define PGSTAT_NUM_PROGRESS_PARAM 12 /* ---------- * Shared-memory data structures diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h index 3d705faba5c..4f2872de35f 100644 --- a/src/include/storage/lmgr.h +++ b/src/include/storage/lmgr.h @@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel, extern bool ConditionalXactLockTableWait(TransactionId xid); /* Lock VXIDs, specified by conflicting locktags */ -extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode); -extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode); +extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress); +extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress); /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */ extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid); diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h index 16b927cb801..e117b391774 100644 --- a/src/include/storage/lock.h +++ b/src/include/storage/lock.h @@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode); extern bool LockHasWaiters(const LOCKTAG *locktag, LOCKMODE lockmode, bool sessionLock); extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag, - LOCKMODE lockmode); + LOCKMODE lockmode, int *countp); extern void AtPrepare_Locks(void); extern void PostPrepare_Locks(TransactionId xid); extern int LockCheckConflicts(LockMethod lockMethodTable, -- 2.17.1 --3V7upXqbjpZ4EhLz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-report-progress-of-hash-indexes.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v8 1/2] Report progress of CREATE INDEX operations @ 2019-03-25 16:29 Alvaro Herrera <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Alvaro Herrera @ 2019-03-25 16:29 UTC (permalink / raw) --- contrib/amcheck/verify_nbtree.c | 2 +- contrib/bloom/blinsert.c | 2 +- contrib/bloom/blutils.c | 1 + doc/src/sgml/indexam.sgml | 13 ++ doc/src/sgml/monitoring.sgml | 224 ++++++++++++++++++++++- src/backend/access/brin/brin.c | 5 +- src/backend/access/gin/gininsert.c | 2 +- src/backend/access/gin/ginutil.c | 1 + src/backend/access/gist/gist.c | 1 + src/backend/access/gist/gistbuild.c | 2 +- src/backend/access/hash/hash.c | 3 +- src/backend/access/heap/heapam_handler.c | 73 ++++++++ src/backend/access/nbtree/nbtree.c | 9 + src/backend/access/nbtree/nbtsort.c | 61 +++++- src/backend/access/nbtree/nbtutils.c | 24 +++ src/backend/access/spgist/spginsert.c | 2 +- src/backend/access/spgist/spgutils.c | 1 + src/backend/catalog/index.c | 67 ++++++- src/backend/catalog/system_views.sql | 27 +++ src/backend/commands/indexcmds.c | 72 +++++++- src/backend/storage/ipc/standby.c | 2 +- src/backend/storage/lmgr/lmgr.c | 46 ++++- src/backend/storage/lmgr/lock.c | 7 +- src/backend/utils/adt/amutils.c | 23 +++ src/backend/utils/adt/pgstatfuncs.c | 2 + src/include/access/amapi.h | 4 + src/include/access/genam.h | 1 + src/include/access/nbtree.h | 11 ++ src/include/access/tableam.h | 10 + src/include/catalog/pg_proc.dat | 10 +- src/include/commands/progress.h | 37 +++- src/include/pgstat.h | 5 +- src/include/storage/lmgr.h | 4 +- src/include/storage/lock.h | 2 +- src/test/regress/expected/rules.out | 30 ++- 35 files changed, 740 insertions(+), 46 deletions(-) diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c index 9ecb1999e34..b55178328c8 100644 --- a/contrib/amcheck/verify_nbtree.c +++ b/contrib/amcheck/verify_nbtree.c @@ -566,7 +566,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace, RelationGetRelationName(state->rel), RelationGetRelationName(state->heaprel)); - table_index_build_scan(state->heaprel, state->rel, indexinfo, true, + table_index_build_scan(state->heaprel, state->rel, indexinfo, true, false, bt_tuple_present_callback, (void *) state, scan); ereport(DEBUG1, diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c index 1b8df7e1e84..48f35d39990 100644 --- a/contrib/bloom/blinsert.c +++ b/contrib/bloom/blinsert.c @@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo) initCachedPage(&buildstate); /* Do the heap scan */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, bloomBuildCallback, (void *) &buildstate, NULL); diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c index d078dfbd469..ee3bd562748 100644 --- a/contrib/bloom/blutils.c +++ b/contrib/bloom/blutils.c @@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = blcostestimate; amroutine->amoptions = bloptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = blvalidate; amroutine->ambeginscan = blbeginscan; amroutine->amrescan = blrescan; diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml index b56d3b3daa1..ff8290da9ff 100644 --- a/doc/src/sgml/indexam.sgml +++ b/doc/src/sgml/indexam.sgml @@ -127,6 +127,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; @@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno, <para> <programlisting> +char * +ambuildphasename (int64 phasenum); +</programlisting> + Return the textual name of the given build phase number. + The phase numbers are those reported during an index build via the + <function>pgstat_progress_update_param</function> interface. + The phase names are then exposed in the + <structname>pg_stat_progress_create_index</structname> view. + </para> + + <para> +<programlisting> bool amvalidate (Oid opclassoid); </programlisting> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index f1df14bdea8..8f5bbbb79e5 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -336,6 +336,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </entry> </row> + <row> + <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry> + <entry>One row for each backend running <command>CREATE INDEX</command>, showing + current progress. + See <xref linkend='create-index-progress-reporting'/>. + </entry> + </row> + <row> <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry> <entry>One row for each backend (including autovacuum worker processes) running @@ -3403,10 +3411,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <para> <productname>PostgreSQL</productname> has the ability to report the progress of certain commands during command execution. Currently, the only commands - which support progress reporting are <command>VACUUM</command> and + which support progress reporting are <command>CREATE INDEX</command>, + <command>VACUUM</command> and <command>CLUSTER</command>. This may be expanded in the future. </para> + <sect2 id="create-index-progress-reporting"> + <title>CREATE INDEX Progress Reporting</title> + + <para> + Whenever <command>CREATE INDEX</command> is running, the + <structname>pg_stat_progress_create_index</structname> view will contain + one row for each backend that is currently creating indexes. The tables + below describe the information that will be reported and provide information + about how to interpret it. + </para> + + <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index"> + <title><structname>pg_stat_progress_create_index</structname> View</title> + <tgroup cols="3"> + <thead> + <row> + <entry>Column</entry> + <entry>Type</entry> + <entry>Description</entry> + </row> + </thead> + + <tbody> + <row> + <entry><structfield>pid</structfield></entry> + <entry><type>integer</type></entry> + <entry>Process ID of backend.</entry> + </row> + <row> + <entry><structfield>datid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>datname</structfield></entry> + <entry><type>name</type></entry> + <entry>Name of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>relid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the table on which the index is being created.</entry> + </row> + <row> + <entry><structfield>phase</structfield></entry> + <entry><type>text</type></entry> + <entry> + Current processing phase of index creation. See <xref linkend='create-index-phases'/>. + </entry> + </row> + <row> + <entry><structfield>lockers_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of lockers to wait for, when applicable. + </entry> + </row> + <row> + <entry><structfield>lockers_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of lockers already waited for. + </entry> + </row> + <row> + <entry><structfield>current_locked_pid</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Process ID of the locker currently being waited for. + </entry> + </row> + <row> + <entry><structfield>blocks_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of blocks to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>blocks_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of blocks already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of tuples to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of tuples already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>partitions_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + total number of partitions on which the index is to be created. + </entry> + </row> + <row> + <entry><structfield>partitions_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + number of partitions on which the index has been completed. + </entry> + </row> + </tbody> + </tgroup> + </table> + + <table id="create-index-phases"> + <title>CREATE INDEX phases</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Phase</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry><literal>initializing</literal></entry> + <entry> + <command>CREATE INDEX</command> is preparing to create the index. This + phase is expected to be very brief. + </entry> + </row> + <row> + <entry><literal>waiting for old snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>building index</literal></entry> + <entry> + The index is being built by the access method-specific code. In this phase, + access methods that support progress reporting fill in their own progress data, + and the subphase is indicated in this column. Typically, + <structname>blocks_total</structname> and <structname>blocks_done</structname> + will contain progress data, as well as potentially + <structname>tuples_total</structname> and <structname>tuples_done</structname>. + </entry> + </row> + <row> + <entry><literal>waiting for writer snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially write into the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>validating index scan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching + for tuples that need to be validated. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the index) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>sorting index scan results</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the + previous phase. + </entry> + </row> + <row> + <entry><literal>validating index heapscan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the table + to validate the index tuples collected in the previous two phases. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the table) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>waiting for reader snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. This + phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + </tbody> + </tgroup> + </table> + + </sect2> + <sect2 id="vacuum-progress-reporting"> <title>VACUUM Progress Reporting</title> diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index 6e96d24ca22..54273f754f6 100644 --- a/src/backend/access/brin/brin.c +++ b/src/backend/access/brin/brin.c @@ -112,6 +112,7 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = brincostestimate; amroutine->amoptions = brinoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = brinvalidate; amroutine->ambeginscan = brinbeginscan; amroutine->amrescan = brinrescan; @@ -719,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Now scan the relation. No syncscan allowed here because we want the * heap blocks in physical order. */ - reltuples = table_index_build_scan(heap, index, indexInfo, false, + reltuples = table_index_build_scan(heap, index, indexInfo, false, false, brinbuildCallback, (void *) state, NULL); /* process the final batch */ @@ -1236,7 +1237,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel, * cases. */ state->bs_currRangeStart = heapBlk; - table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, + table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, false, heapBlk, scanNumBlks, brinbuildCallback, (void *) state, NULL); diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c index b02f69b0dcb..b4b0213f76f 100644 --- a/src/backend/access/gin/gininsert.c +++ b/src/backend/access/gin/gininsert.c @@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Do the heap scan. We disallow sync scan here because dataPlaceToPage * prefers to receive tuples in TID order. */ - reltuples = table_index_build_scan(heap, index, indexInfo, false, + reltuples = table_index_build_scan(heap, index, indexInfo, false, false, ginBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c index afc20232ace..d2360eeafb0 100644 --- a/src/backend/access/gin/ginutil.c +++ b/src/backend/access/gin/ginutil.c @@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gincostestimate; amroutine->amoptions = ginoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = ginvalidate; amroutine->ambeginscan = ginbeginscan; amroutine->amrescan = ginrescan; diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 2fddb23496d..f44c922b5d6 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gistcostestimate; amroutine->amoptions = gistoptions; amroutine->amproperty = gistproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = gistvalidate; amroutine->ambeginscan = gistbeginscan; amroutine->amrescan = gistrescan; diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index 3652fde5bb1..771bd2962a7 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) /* * Do the heap scan. */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, gistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c index 5cc12a17130..6b38dd9c0e9 100644 --- a/src/backend/access/hash/hash.c +++ b/src/backend/access/hash/hash.c @@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = hashcostestimate; amroutine->amoptions = hashoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = hashvalidate; amroutine->ambeginscan = hashbeginscan; amroutine->amrescan = hashrescan; @@ -160,7 +161,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo) buildstate.heapRel = heap; /* do the heap scan */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, hashbuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 39eab1e28d9..dec653d3b4d 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -54,6 +54,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, static const TableAmRoutine heapam_methods; +static BlockNumber heapscan_get_blocks_done(HeapScanDesc hscan); + + /* ------------------------------------------------------------------------ * Slot related callbacks for heap AM * ------------------------------------------------------------------------ @@ -927,6 +930,7 @@ heapam_index_build_range_scan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -947,6 +951,7 @@ heapam_index_build_range_scan(Relation heapRelation, Snapshot snapshot; bool need_unregister_snapshot = false; TransactionId OldestXmin; + BlockNumber previous_blkno = InvalidBlockNumber; BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; @@ -1017,6 +1022,14 @@ heapam_index_build_range_scan(Relation heapRelation, NULL, /* scan key */ true, /* buffer access strategy OK */ allow_sync); /* syncscan OK? */ + + if (progress) + { + hscan = (HeapScanDesc) scan; + Assert(hscan->rs_numblocks == InvalidBlockNumber); + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + hscan->rs_nblocks); + } } else { @@ -1066,6 +1079,19 @@ heapam_index_build_range_scan(Relation heapRelation, CHECK_FOR_INTERRUPTS(); + /* Report scan progress, if asked to. */ + if (progress) + { + BlockNumber blocks_done = heapscan_get_blocks_done(hscan); + + if (blocks_done != previous_blkno) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blocks_done); + previous_blkno = blocks_done; + } + } + /* * When dealing with a HOT-chain of updated tuples, we want to index * the values of the live tuple (if any), but index it under the TID @@ -1440,6 +1466,7 @@ heapam_index_validate_scan(Relation heapRelation, BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; bool in_index[MaxHeapTuplesPerPage]; + BlockNumber previous_blkno = InvalidBlockNumber; /* state variables for the merge */ ItemPointer indexcursor = NULL; @@ -1480,6 +1507,9 @@ heapam_index_validate_scan(Relation heapRelation, false); /* syncscan not OK */ hscan = (HeapScanDesc) scan; + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + hscan->rs_nblocks); + /* * Scan all tuples matching the snapshot. */ @@ -1493,6 +1523,14 @@ heapam_index_validate_scan(Relation heapRelation, state->htups += 1; + if ((previous_blkno == InvalidBlockNumber) || + (hscan->rs_cblock != previous_blkno)) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + hscan->rs_cblock); + previous_blkno = hscan->rs_cblock; + } + /* * As commented in table_index_build_scan, we should index heap-only * tuples under the TIDs of their root tuples; so when we advance onto @@ -1653,6 +1691,41 @@ heapam_index_validate_scan(Relation heapRelation, indexInfo->ii_PredicateState = NULL; } +/* + * Return the number of blocks that have been read by this scan since + * starting. This is not 100% accurate: in a parallel scan, the workers + * can be concurrently reading blocks further ahead than what we report. + */ +static BlockNumber +heapscan_get_blocks_done(HeapScanDesc hscan) +{ + BlockNumber startblock; + BlockNumber blocks_done; + + if (hscan->rs_base.rs_parallel != NULL) + { + ParallelBlockTableScanDesc bpscan; + + bpscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel; + startblock = bpscan->phs_startblock; + } + else + startblock = hscan->rs_startblock; + + /* + * Might have wrapped around the end of the relation, if startblock was + * not zero. + */ + if (hscan->rs_cblock > startblock) + blocks_done = hscan->rs_cblock - startblock; + else + blocks_done = hscan->rs_nblocks - startblock + + hscan->rs_cblock; + + return blocks_done; +} + + /* ---------------------------------------------------------------------------- * Helper functions for the above. diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index ac6f1eb3423..7370379c6a1 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -22,6 +22,7 @@ #include "access/nbtxlog.h" #include "access/relscan.h" #include "access/xlog.h" +#include "commands/progress.h" #include "commands/vacuum.h" #include "miscadmin.h" #include "nodes/execnodes.h" @@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = btcostestimate; amroutine->amoptions = btoptions; amroutine->amproperty = btproperty; + amroutine->ambuildphasename = btbuildphasename; amroutine->amvalidate = btvalidate; amroutine->ambeginscan = btbeginscan; amroutine->amrescan = btrescan; @@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (needLock) UnlockRelationForExtension(rel, ExclusiveLock); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + num_pages); + /* Quit if we've scanned the whole relation */ if (blkno >= num_pages) break; @@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, for (; blkno < num_pages; blkno++) { btvacuumpage(&vstate, blkno, blkno); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blkno); } } diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index a8a7b792672..b889ac6de79 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -66,6 +66,7 @@ #include "access/xlog.h" #include "access/xloginsert.h" #include "catalog/index.h" +#include "commands/progress.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/smgr.h" @@ -298,7 +299,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate, static void _bt_leader_participate_as_worker(BTBuildState *buildstate); static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem); + Sharedsort *sharedsort2, int sortmem, + bool progress); /* @@ -394,6 +396,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Save as primary spool */ buildstate->spool = btspool; + /* Report heap scan phase started */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN); + /* Attempt to launch parallel worker scan when required */ if (indexInfo->ii_ParallelWorkers > 0) _bt_begin_parallel(buildstate, indexInfo->ii_Concurrent, @@ -480,13 +486,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Fill spool using either serial or parallel heap scan */ if (!buildstate->btleader) - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, true, _bt_build_callback, (void *) buildstate, NULL); else reltuples = _bt_parallel_heapscan(buildstate, &indexInfo->ii_BrokenHotChain); + /* + * Set the progress target for the next phase. Reset the block number + * values set by IndexBuildHeapScan + */ + { + const int index[] = { + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE + }; + const int64 val[] = { + buildstate->indtuples, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } + /* okay, all heap tuples are spooled */ if (buildstate->spool2 && !buildstate->havedead) { @@ -535,9 +559,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) } #endif /* BTREE_BUILD_STATS */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_1); tuplesort_performsort(btspool->sortstate); if (btspool2) + { + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_2); tuplesort_performsort(btspool2->sortstate); + } wstate.heap = btspool->heap; wstate.index = btspool->index; @@ -554,6 +584,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) wstate.btws_pages_written = 0; wstate.btws_zeropage = NULL; /* until needed */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_LEAF_LOAD); _bt_load(&wstate, btspool, btspool2); } @@ -1098,6 +1130,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) int i, keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index); SortSupport sortKeys; + long tuples_done = 0L; if (merge) { @@ -1202,6 +1235,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) _bt_buildadd(wstate, state, itup2); itup2 = tuplesort_getindextuple(btspool2->sortstate, true); } + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } pfree(sortKeys); } @@ -1216,6 +1253,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) state = _bt_pagestate(wstate, 0); _bt_buildadd(wstate, state, itup); + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } } @@ -1352,6 +1393,11 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request) ParallelTableScanFromBTShared(btshared), snapshot); + /* Report total number of blocks to scan */ + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + ((ParallelBlockTableScanDescData *) + ParallelTableScanFromBTShared(btshared))->phs_nblocks); + /* * Store shared tuplesort-private state, for which we reserved space. * Then, initialize opaque state using tuplesort routine. @@ -1528,7 +1574,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate) /* Perform work common to all participants */ _bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared, btleader->sharedsort, btleader->sharedsort2, - sortmem); + sortmem, true); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1619,7 +1665,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) /* Perform sorting of spool, and possibly a spool2 */ sortmem = maintenance_work_mem / btshared->scantuplesortstates; _bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort, - sharedsort2, sortmem); + sharedsort2, sortmem, false); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1648,7 +1694,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem) + Sharedsort *sharedsort2, int sortmem, bool progress) { SortCoordinate coordinate; BTBuildState buildstate; @@ -1705,9 +1751,10 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, /* Join parallel scan */ indexInfo = BuildIndexInfo(btspool->index); indexInfo->ii_Concurrent = btshared->isconcurrent; - scan = table_beginscan_parallel(btspool->heap, ParallelTableScanFromBTShared(btshared)); + scan = table_beginscan_parallel(btspool->heap, + ParallelTableScanFromBTShared(btshared)); reltuples = table_index_build_scan(btspool->heap, btspool->index, indexInfo, - true, _bt_build_callback, + true, progress, _bt_build_callback, (void *) &buildstate, scan); /* diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c index 92b8b5f134d..9e8a7bd3d9a 100644 --- a/src/backend/access/nbtree/nbtutils.c +++ b/src/backend/access/nbtree/nbtutils.c @@ -20,6 +20,7 @@ #include "access/nbtree.h" #include "access/reloptions.h" #include "access/relscan.h" +#include "commands/progress.h" #include "miscadmin.h" #include "utils/array.h" #include "utils/datum.h" @@ -2048,6 +2049,29 @@ btproperty(Oid index_oid, int attno, } } +/* + * btbuildphasename() -- Return name of index build phase. + */ +char * +btbuildphasename(int64 phasenum) +{ + switch (phasenum) + { + case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE: + return "initializing"; + case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN: + return "table scan"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_1: + return "sorting tuples, spool 1"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_2: + return "sorting tuples, spool 2"; + case PROGRESS_BTREE_PHASE_LEAF_LOAD: + return "btree tuple loading"; + default: + return NULL; + } +} + /* * _bt_truncate() -- create tuple without unneeded suffix attributes. * diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c index 390ad9ac51f..282d6998cf0 100644 --- a/src/backend/access/spgist/spginsert.c +++ b/src/backend/access/spgist/spginsert.c @@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo) "SP-GiST build temporary context", ALLOCSET_DEFAULT_SIZES); - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, spgistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c index 8e63c1fad25..45472db147b 100644 --- a/src/backend/access/spgist/spgutils.c +++ b/src/backend/access/spgist/spgutils.c @@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = spgcostestimate; amroutine->amoptions = spgoptions; amroutine->amproperty = spgproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = spgvalidate; amroutine->ambeginscan = spgbeginscan; amroutine->amrescan = spgrescan; diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 0d9d405c548..6c6818e891c 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -51,9 +51,9 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" -#include "commands/tablecmds.h" #include "commands/event_trigger.h" #include "commands/progress.h" +#include "commands/tablecmds.h" #include "commands/trigger.h" #include "executor/executor.h" #include "miscadmin.h" @@ -2047,7 +2047,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode) * to acquire an exclusive lock on our table. The lock code will * detect deadlock and error out properly. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* Finish invalidation of index and mark it as dead */ index_concurrently_set_dead(heapId, indexId); @@ -2063,7 +2063,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode) * Wait till every transaction that saw the old index state has * finished. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * Re-open relations to allow us to complete our actions. @@ -2712,6 +2712,25 @@ index_build(Relation heapRelation, save_sec_context | SECURITY_RESTRICTED_OPERATION); save_nestlevel = NewGUCNestLevel(); + /* Set up initial progress report status */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_BUILD, + PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE, + 0, 0, 0, 0 + }; + + pgstat_progress_update_multi_param(6, index, val); + } + /* * Call the access method's build procedure */ @@ -3000,6 +3019,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) int save_sec_context; int save_nestlevel; + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN, + 0, 0, 0, 0 + }; + pgstat_progress_update_multi_param(5, index, val); + } + /* Open and lock the parent heap relation */ heapRelation = table_open(heapId, ShareUpdateExclusiveLock); /* And the target index relation */ @@ -3030,6 +3064,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) */ ivinfo.index = indexRelation; ivinfo.analyze_only = false; + ivinfo.report_progress = true; /* XXX only for btree? */ ivinfo.estimated_count = true; ivinfo.message_level = DEBUG2; ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples; @@ -3047,15 +3082,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) NULL, false); state.htups = state.itups = state.tups_inserted = 0; + /* ambulkdelete updates progress metrics */ (void) index_bulk_delete(&ivinfo, NULL, validate_index_callback, (void *) &state); /* Execute the sort */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } tuplesort_performsort(state.tuplesort); /* - * Now scan the heap and "merge" it with the index + * Now scan the heap and "merge" it with the index. */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE + }; + + pgstat_progress_update_multi_param(1, index, val); + } table_index_validate_scan(heapRelation, indexRelation, indexInfo, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index b89df70653e..acf277473c1 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -934,6 +934,33 @@ CREATE VIEW pg_stat_progress_cluster AS FROM pg_stat_get_progress_info('CLUSTER') AS S LEFT JOIN pg_database D ON S.datid = D.oid; +CREATE VIEW pg_stat_progress_create_index AS + SELECT + S.pid AS pid, S.datid AS datid, D.datname AS datname, + S.relid AS relid, + CASE S.param10 WHEN 0 THEN 'initializing' + WHEN 1 THEN 'waiting for old snapshots' + WHEN 2 THEN 'building index' || + COALESCE((': ' || pg_indexam_progress_phasename(S.param9::oid, S.param11)), + '') + WHEN 3 THEN 'waiting for writer snapshots' + WHEN 4 THEN 'index validation: scan index' + WHEN 5 THEN 'index validation: sort index scan results' + WHEN 6 THEN 'index validation: scan heap' + WHEN 7 THEN 'waiting for reader snapshots' + END as phase, + S.param4 AS lockers_total, + S.param5 AS lockers_done, + S.param6 AS current_locker_pid, + S.param16 AS blocks_total, + S.param17 AS blocks_done, + S.param12 AS tuples_total, + S.param13 AS tuples_done, + S.param14 AS partitions_total, + S.param15 AS partitions_done + FROM pg_stat_get_progress_info('CREATE INDEX') AS S + LEFT JOIN pg_database D ON S.datid = D.oid; + CREATE VIEW pg_user_mappings AS SELECT U.oid AS umid, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index ea07be69dbd..905e48d458c 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -36,6 +36,7 @@ #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" +#include "commands/progress.h" #include "commands/tablecmds.h" #include "commands/tablespace.h" #include "mb/pg_wchar.h" @@ -47,10 +48,12 @@ #include "parser/parse_func.h" #include "parser/parse_oper.h" #include "partitioning/partdesc.h" +#include "pgstat.h" #include "rewrite/rewriteManip.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/acl.h" #include "utils/builtins.h" #include "utils/fmgroids.h" @@ -334,7 +337,7 @@ CheckIndexCompatible(Oid oldId, * doesn't show up in the output, we know we can forget about it. */ static void -WaitForOlderSnapshots(TransactionId limitXmin) +WaitForOlderSnapshots(TransactionId limitXmin, bool progress) { int n_old_snapshots; int i; @@ -343,6 +346,8 @@ WaitForOlderSnapshots(TransactionId limitXmin) old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false, PROC_IS_AUTOVACUUM | PROC_IN_VACUUM, &n_old_snapshots); + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots); for (i = 0; i < n_old_snapshots; i++) { @@ -378,7 +383,19 @@ WaitForOlderSnapshots(TransactionId limitXmin) } if (VirtualTransactionIdIsValid(old_snapshots[i])) + { + if (progress) + { + PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId); + + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); + } VirtualXactLock(old_snapshots[i], true); + } + + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1); } } @@ -452,6 +469,15 @@ DefineIndex(Oid relationId, Snapshot snapshot; int i; + + /* + * Start progress report. If we're building a partition, this was already + * done. + */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX, + relationId); + /* * count key attributes in index */ @@ -668,6 +694,9 @@ DefineIndex(Oid relationId, accessMethodId = accessMethodForm->oid; amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler); + pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID, + accessMethodId); + if (stmt->unique && !amRoutine->amcanunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -948,6 +977,11 @@ DefineIndex(Oid relationId, if (!OidIsValid(indexRelationId)) { table_close(rel, NoLock); + + /* If this is the top-level index, we're done */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -973,6 +1007,9 @@ DefineIndex(Oid relationId, TupleDesc parentDesc; Oid *opfamOids; + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL, + nparts); + memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts); parentDesc = CreateTupleDescCopy(RelationGetDescr(rel)); @@ -1122,6 +1159,8 @@ DefineIndex(Oid relationId, skip_build, quiet); } + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, + i + 1); pfree(attmap); } @@ -1156,6 +1195,8 @@ DefineIndex(Oid relationId, * Indexes on partitioned tables are not themselves built, so we're * done here. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); return address; } @@ -1163,6 +1204,11 @@ DefineIndex(Oid relationId, { /* Close the heap and we're done, in the non-concurrent case */ table_close(rel, NoLock); + + /* If this is the top-level index, we're done. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -1214,7 +1260,9 @@ DefineIndex(Oid relationId, * exclusive lock on our table. The lock code will detect deadlock and * error out properly. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_1); + WaitForLockers(heaplocktag, ShareLock, true); /* * At this moment we are sure that there are no transactions with the @@ -1255,7 +1303,9 @@ DefineIndex(Oid relationId, * We once again wait until no transaction can have the table open with * the index marked as read-only for updates. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_2); + WaitForLockers(heaplocktag, ShareLock, true); /* * Now take the "reference snapshot" that will be used by validate_index() @@ -1312,7 +1362,9 @@ DefineIndex(Oid relationId, * before the reference snap was taken, we have to wait out any * transactions that might have older snapshots. */ - WaitForOlderSnapshots(limitXmin); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_3); + WaitForOlderSnapshots(limitXmin, true); /* * Index can now be marked valid -- update its pg_index entry @@ -1334,6 +1386,8 @@ DefineIndex(Oid relationId, */ UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock); + pgstat_progress_end_command(); + return address; } @@ -2913,7 +2967,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * DefineIndex() for more details. */ - WaitForLockersMultiple(lockTags, ShareLock); + WaitForLockersMultiple(lockTags, ShareLock, false); CommitTransactionCommand(); forboth(lc, indexIds, lc2, newIndexIds) @@ -2955,7 +3009,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * for more details. */ - WaitForLockersMultiple(lockTags, ShareLock); + WaitForLockersMultiple(lockTags, ShareLock, false); CommitTransactionCommand(); foreach(lc, newIndexIds) @@ -3003,7 +3057,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * before the reference snap was taken, we have to wait out any * transactions that might have older snapshots. */ - WaitForOlderSnapshots(limitXmin); + WaitForOlderSnapshots(limitXmin, false); CommitTransactionCommand(); } @@ -3074,7 +3128,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * index_drop() for more details. */ - WaitForLockersMultiple(lockTags, AccessExclusiveLock); + WaitForLockersMultiple(lockTags, AccessExclusiveLock, false); foreach(lc, indexIds) { @@ -3096,7 +3150,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * Drop the old indexes. */ - WaitForLockersMultiple(lockTags, AccessExclusiveLock); + WaitForLockersMultiple(lockTags, AccessExclusiveLock, false); PushActiveSnapshot(GetTransactionSnapshot()); diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index cd56dca3aef..215f1463bb1 100644 --- a/src/backend/storage/ipc/standby.c +++ b/src/backend/storage/ipc/standby.c @@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag) */ VirtualTransactionId *backends; - backends = GetLockConflicts(&locktag, AccessExclusiveLock); + backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL); ResolveRecoveryConflictWithVirtualXIDs(backends, PROCSIG_RECOVERY_CONFLICT_LOCK); } diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c index e688ba81170..0b04b093782 100644 --- a/src/backend/storage/lmgr/lmgr.c +++ b/src/backend/storage/lmgr/lmgr.c @@ -19,9 +19,12 @@ #include "access/transam.h" #include "access/xact.h" #include "catalog/catalog.h" +#include "commands/progress.h" #include "miscadmin.h" +#include "pgstat.h" #include "storage/lmgr.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/inval.h" @@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg) * after we obtained our initial list of lockers, we will not wait for them. */ void -WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) +WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress) { List *holders = NIL; ListCell *lc; + int total = 0; + int done = 0; /* Done if no locks to wait for */ if (list_length(locktags) == 0) @@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) foreach(lc, locktags) { LOCKTAG *locktag = lfirst(lc); + int count; - holders = lappend(holders, GetLockConflicts(locktag, lockmode)); + holders = lappend(holders, + GetLockConflicts(locktag, lockmode, + progress ? &count : NULL)); + total += count; } + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total); + /* * Note: GetLockConflicts() never reports our own xid, hence we need not * check for that. Also, prepared xacts are not reported, which is fine @@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) while (VirtualTransactionIdIsValid(*lockholders)) { + /* + * If requested, publish who we're going to wait for. This is not + * 100% accurate if they're already gone, but we don't care. + */ + if (progress) + { + PGPROC *holder = BackendIdGetProc(lockholders->backendId); + + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); + } VirtualXactLock(*lockholders, true); lockholders++; + + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done); } } + if (progress) + { + const int index[] = { + PROGRESS_WAITFOR_TOTAL, + PROGRESS_WAITFOR_DONE, + PROGRESS_WAITFOR_CURRENT_PID + }; + const int64 values[] = { + 0, 0, 0 + }; + pgstat_progress_update_multi_param(3, index, values); + } list_free_deep(holders); } @@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) * Same as WaitForLockersMultiple, for a single lock tag. */ void -WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode) +WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress) { List *l; l = list_make1(&heaplocktag); - WaitForLockersMultiple(l, lockmode); + WaitForLockersMultiple(l, lockmode, progress); list_free(l); } diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c index 78fdbd6ff88..c8958766f1e 100644 --- a/src/backend/storage/lmgr/lock.c +++ b/src/backend/storage/lmgr/lock.c @@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * xacts merely awaiting such a lock are NOT reported. * * The result array is palloc'd and is terminated with an invalid VXID. + * *countp, if not null, is updated to the number of items set. * * Of course, the result could be out of date by the time it's returned, * so use of this function has to be thought about carefully. @@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * uses of the result. */ VirtualTransactionId * -GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) +GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp) { static VirtualTransactionId *vxids; LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid; @@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) LWLockRelease(partitionLock); vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } @@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c index 060ffe501ec..e81d6cc0562 100644 --- a/src/backend/utils/adt/amutils.c +++ b/src/backend/utils/adt/amutils.c @@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS) return indexam_property(fcinfo, propname, InvalidOid, relid, attno); } + +/* + * Return the name of the given phase, as used for progress reporting by the + * given AM. + */ +Datum +pg_indexam_progress_phasename(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + int32 phasenum = PG_GETARG_INT32(1); + IndexAmRoutine *routine; + char *name; + + routine = GetIndexAmRoutineByAmId(amoid, true); + if (routine == NULL || !routine->ambuildphasename) + PG_RETURN_NULL(); + + name = routine->ambuildphasename(phasenum); + if (!name) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(CStringGetTextDatum(name)); +} diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 90a817a25c5..7c2afe64272 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -470,6 +470,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS) cmdtype = PROGRESS_COMMAND_VACUUM; else if (pg_strcasecmp(cmd, "CLUSTER") == 0) cmdtype = PROGRESS_COMMAND_CLUSTER; + else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0) + cmdtype = PROGRESS_COMMAND_CREATE_INDEX; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index 653ddc976ba..09a7404267c 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +/* name of phase as used in progress reporting */ +typedef char *(*ambuildphasename_function) (int64 phasenum); + /* validate definition of an opclass for this AM */ typedef bool (*amvalidate_function) (Oid opclassoid); @@ -213,6 +216,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 70c7351a08c..9717183ef23 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo { Relation index; /* the index being vacuumed */ bool analyze_only; /* ANALYZE (without any actual vacuum) */ + bool report_progress; /* emit progress.h status reports */ bool estimated_count; /* num_heap_tuples is an estimate */ int message_level; /* ereport level for progress messages */ double num_heap_tuples; /* tuples remaining in heap */ diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h index 473c6f29185..bc0f3027629 100644 --- a/src/include/access/nbtree.h +++ b/src/include/access/nbtree.h @@ -672,6 +672,16 @@ typedef BTScanOpaqueData *BTScanOpaque; #define SK_BT_DESC (INDOPTION_DESC << SK_BT_INDOPTION_SHIFT) #define SK_BT_NULLS_FIRST (INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT) +/* + * Constant definition for progress reporting. Phase numbers must match + * btbuildphasename. + */ +/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */ +#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN 2 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_1 3 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_2 4 +#define PROGRESS_BTREE_PHASE_LEAF_LOAD 5 + /* * external entry points for btree, in nbtree.c */ @@ -785,6 +795,7 @@ extern bytea *btoptions(Datum reloptions, bool validate); extern bool btproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +extern char *btbuildphasename(int64 phasenum); extern IndexTuple _bt_truncate(Relation rel, IndexTuple lastleft, IndexTuple firstright, BTScanInsert itup_key); extern int _bt_keep_natts_fast(Relation rel, IndexTuple lastleft, diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index c571f8a8994..a5565d1ef43 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -424,6 +424,7 @@ typedef struct TableAmRoutine struct IndexInfo *index_nfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber end_blockno, IndexBuildCallback callback, @@ -1095,6 +1096,11 @@ table_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, * so here because the AM might reject some of the tuples for its own reasons, * such as being unable to store NULLs. * + * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go + * along. Also, if that flag is true and a scan descriptor is not passed (ie. + * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is + * updated at the beginning. For parallel scans, caller is expected to have + * set the total number of blocks prior to calling this function. * * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect * any potentially broken HOT chains. Currently, we set this if there are any @@ -1108,6 +1114,7 @@ table_index_build_scan(Relation heap_rel, Relation index_rel, struct IndexInfo *index_nfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, TableScanDesc scan) @@ -1117,6 +1124,7 @@ table_index_build_scan(Relation heap_rel, index_nfo, allow_sync, false, + progress, 0, InvalidBlockNumber, callback, @@ -1140,6 +1148,7 @@ table_index_build_range_scan(Relation heap_rel, struct IndexInfo *index_nfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -1151,6 +1160,7 @@ table_index_build_range_scan(Relation heap_rel, index_nfo, allow_sync, anyvisible, + progress, start_blockno, numblocks, callback, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index eac909109c5..a7050edca09 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -924,6 +924,10 @@ proname => 'pg_index_column_has_property', provolatile => 's', prorettype => 'bool', proargtypes => 'regclass int4 text', prosrc => 'pg_index_column_has_property' }, +{ oid => '676', descr => 'return name of given index build phase', + proname => 'pg_indexam_progress_phasename', provolatile => 'i', + prorettype => 'text', proargtypes => 'oid int8', + prosrc => 'pg_indexam_progress_phasename' }, { oid => '339', proname => 'poly_same', prorettype => 'bool', @@ -5122,9 +5126,9 @@ proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't', provolatile => 's', proparallel => 'r', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', - proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}', + proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12,param13,param14,param15,param16,param17,param18,param19,param20}', prosrc => 'pg_stat_get_progress_info' }, { oid => '3099', descr => 'statistics: information about currently active replication', diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h index 04542d9e923..079303cff46 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -44,7 +44,7 @@ #define PROGRESS_CLUSTER_HEAP_BLKS_SCANNED 6 #define PROGRESS_CLUSTER_INDEX_REBUILD_COUNT 7 -/* Phases of cluster (as dvertised via PROGRESS_CLUSTER_PHASE) */ +/* Phases of cluster (as advertised via PROGRESS_CLUSTER_PHASE) */ #define PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP 1 #define PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP 2 #define PROGRESS_CLUSTER_PHASE_SORT_TUPLES 3 @@ -57,4 +57,39 @@ #define PROGRESS_CLUSTER_COMMAND_CLUSTER 1 #define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL 2 +/* Progress parameters for CREATE INDEX */ +/* 3, 4 and 5 reserved for "waitfor" metrics */ +#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID 8 +#define PROGRESS_CREATEIDX_PHASE 9 /* AM-agnostic phase # */ +#define PROGRESS_CREATEIDX_SUBPHASE 10 /* phase # filled by AM */ +#define PROGRESS_CREATEIDX_TUPLES_TOTAL 11 +#define PROGRESS_CREATEIDX_TUPLES_DONE 12 +#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL 13 +#define PROGRESS_CREATEIDX_PARTITIONS_DONE 14 +/* 15 and 16 reserved for "block number" metrics */ + +/* Phases of CREATE INDEX (as advertised via PROGRESS_CREATEIDX_PHASE) */ +#define PROGRESS_CREATEIDX_PHASE_WAIT_1 1 +#define PROGRESS_CREATEIDX_PHASE_BUILD 2 +#define PROGRESS_CREATEIDX_PHASE_WAIT_2 3 +#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN 4 +#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN 5 +#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE 6 +#define PROGRESS_CREATEIDX_PHASE_WAIT_3 7 + +/* + * Subphases of CREATE INDEX, for index_build. + */ +#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE 1 +/* Additional phases are defined by each AM */ + +/* Lock holder wait counts */ +#define PROGRESS_WAITFOR_TOTAL 3 +#define PROGRESS_WAITFOR_DONE 4 +#define PROGRESS_WAITFOR_CURRENT_PID 5 + +/* Block numbers in a generic relation scan */ +#define PROGRESS_SCAN_BLOCKS_TOTAL 15 +#define PROGRESS_SCAN_BLOCKS_DONE 16 + #endif diff --git a/src/include/pgstat.h b/src/include/pgstat.h index c080fa6388f..53d4a9c4319 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -951,10 +951,11 @@ typedef enum ProgressCommandType { PROGRESS_COMMAND_INVALID, PROGRESS_COMMAND_VACUUM, - PROGRESS_COMMAND_CLUSTER + PROGRESS_COMMAND_CLUSTER, + PROGRESS_COMMAND_CREATE_INDEX } ProgressCommandType; -#define PGSTAT_NUM_PROGRESS_PARAM 10 +#define PGSTAT_NUM_PROGRESS_PARAM 20 /* ---------- * Shared-memory data structures diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h index 3d705faba5c..4f2872de35f 100644 --- a/src/include/storage/lmgr.h +++ b/src/include/storage/lmgr.h @@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel, extern bool ConditionalXactLockTableWait(TransactionId xid); /* Lock VXIDs, specified by conflicting locktags */ -extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode); -extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode); +extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress); +extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress); /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */ extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid); diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h index badf7fd682b..048947c50d4 100644 --- a/src/include/storage/lock.h +++ b/src/include/storage/lock.h @@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode); extern bool LockHasWaiters(const LOCKTAG *locktag, LOCKMODE lockmode, bool sessionLock); extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag, - LOCKMODE lockmode); + LOCKMODE lockmode, int *countp); extern void AtPrepare_Locks(void); extern void PostPrepare_Locks(TransactionId xid); extern int LockCheckConflicts(LockMethod lockMethodTable, diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index d5f309fbfbe..a9a2910fb55 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1856,7 +1856,33 @@ pg_stat_progress_cluster| SELECT s.pid, s.param6 AS heap_blks_total, s.param7 AS heap_blks_scanned, s.param8 AS index_rebuild_count - FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) + LEFT JOIN pg_database d ON ((s.datid = d.oid))); +pg_stat_progress_create_index| SELECT s.pid, + s.datid, + d.datname, + s.relid, + CASE s.param2 + WHEN 0 THEN 'initializing'::text + WHEN 1 THEN 'waiting for old snapshots'::text + WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param1)::oid, s.param3)), ''::text)) + WHEN 3 THEN 'waiting for writer snapshots'::text + WHEN 4 THEN 'index validation: scan index'::text + WHEN 5 THEN 'index validation: sort index scan results'::text + WHEN 6 THEN 'index validation: scan heap'::text + WHEN 7 THEN 'waiting for reader snapshots'::text + ELSE NULL::text + END AS phase, + s.param4 AS lockers_total, + s.param5 AS lockers_done, + s.param6 AS current_locker_pid, + s.param7 AS blocks_total, + s.param8 AS blocks_done, + s.param9 AS tuples_total, + s.param10 AS tuples_done, + s.param11 AS partitions_total, + s.param12 AS partitions_done + FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_progress_vacuum| SELECT s.pid, s.datid, @@ -1878,7 +1904,7 @@ pg_stat_progress_vacuum| SELECT s.pid, s.param5 AS index_vacuum_count, s.param6 AS max_dead_tuples, s.param7 AS num_dead_tuples - FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_replication| SELECT s.pid, s.usesysid, -- 2.17.1 --7AUc2qLy4jB3hD7Z Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v8-0002-report-partial-progress-of-other-index-AMs.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v9 1/2] Report progress of CREATE INDEX operations @ 2019-03-25 16:29 Alvaro Herrera <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Alvaro Herrera @ 2019-03-25 16:29 UTC (permalink / raw) --- contrib/amcheck/verify_nbtree.c | 2 +- contrib/bloom/blinsert.c | 2 +- contrib/bloom/blutils.c | 1 + doc/src/sgml/indexam.sgml | 13 ++ doc/src/sgml/monitoring.sgml | 224 ++++++++++++++++++++++- src/backend/access/brin/brin.c | 5 +- src/backend/access/gin/gininsert.c | 2 +- src/backend/access/gin/ginutil.c | 1 + src/backend/access/gist/gist.c | 1 + src/backend/access/gist/gistbuild.c | 2 +- src/backend/access/hash/hash.c | 3 +- src/backend/access/heap/heapam_handler.c | 107 +++++++++++ src/backend/access/nbtree/nbtree.c | 9 + src/backend/access/nbtree/nbtsort.c | 56 +++++- src/backend/access/nbtree/nbtutils.c | 24 +++ src/backend/access/spgist/spginsert.c | 2 +- src/backend/access/spgist/spgutils.c | 1 + src/backend/catalog/index.c | 67 ++++++- src/backend/catalog/system_views.sql | 27 +++ src/backend/commands/indexcmds.c | 72 +++++++- src/backend/storage/ipc/standby.c | 2 +- src/backend/storage/lmgr/lmgr.c | 46 ++++- src/backend/storage/lmgr/lock.c | 7 +- src/backend/utils/adt/amutils.c | 23 +++ src/backend/utils/adt/pgstatfuncs.c | 2 + src/include/access/amapi.h | 4 + src/include/access/genam.h | 1 + src/include/access/nbtree.h | 11 ++ src/include/access/tableam.h | 10 + src/include/catalog/pg_proc.dat | 10 +- src/include/commands/progress.h | 37 +++- src/include/pgstat.h | 5 +- src/include/storage/lmgr.h | 4 +- src/include/storage/lock.h | 2 +- src/test/regress/expected/rules.out | 30 ++- 35 files changed, 769 insertions(+), 46 deletions(-) diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c index 9d5b2e5be67..591e0a3e46a 100644 --- a/contrib/amcheck/verify_nbtree.c +++ b/contrib/amcheck/verify_nbtree.c @@ -566,7 +566,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace, RelationGetRelationName(state->rel), RelationGetRelationName(state->heaprel)); - table_index_build_scan(state->heaprel, state->rel, indexinfo, true, + table_index_build_scan(state->heaprel, state->rel, indexinfo, true, false, bt_tuple_present_callback, (void *) state, scan); ereport(DEBUG1, diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c index 1b8df7e1e84..48f35d39990 100644 --- a/contrib/bloom/blinsert.c +++ b/contrib/bloom/blinsert.c @@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo) initCachedPage(&buildstate); /* Do the heap scan */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, bloomBuildCallback, (void *) &buildstate, NULL); diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c index d078dfbd469..ee3bd562748 100644 --- a/contrib/bloom/blutils.c +++ b/contrib/bloom/blutils.c @@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = blcostestimate; amroutine->amoptions = bloptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = blvalidate; amroutine->ambeginscan = blbeginscan; amroutine->amrescan = blrescan; diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml index b56d3b3daa1..ff8290da9ff 100644 --- a/doc/src/sgml/indexam.sgml +++ b/doc/src/sgml/indexam.sgml @@ -127,6 +127,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; @@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno, <para> <programlisting> +char * +ambuildphasename (int64 phasenum); +</programlisting> + Return the textual name of the given build phase number. + The phase numbers are those reported during an index build via the + <function>pgstat_progress_update_param</function> interface. + The phase names are then exposed in the + <structname>pg_stat_progress_create_index</structname> view. + </para> + + <para> +<programlisting> bool amvalidate (Oid opclassoid); </programlisting> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index f1df14bdea8..3a007d6871b 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -336,6 +336,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </entry> </row> + <row> + <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry> + <entry>One row for each backend running <command>CREATE INDEX</command>, showing + current progress. + See <xref linkend='create-index-progress-reporting'/>. + </entry> + </row> + <row> <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry> <entry>One row for each backend (including autovacuum worker processes) running @@ -3403,10 +3411,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <para> <productname>PostgreSQL</productname> has the ability to report the progress of certain commands during command execution. Currently, the only commands - which support progress reporting are <command>VACUUM</command> and + which support progress reporting are <command>CREATE INDEX</command>, + <command>VACUUM</command> and <command>CLUSTER</command>. This may be expanded in the future. </para> + <sect2 id="create-index-progress-reporting"> + <title>CREATE INDEX Progress Reporting</title> + + <para> + Whenever <command>CREATE INDEX</command> is running, the + <structname>pg_stat_progress_create_index</structname> view will contain + one row for each backend that is currently creating indexes. The tables + below describe the information that will be reported and provide information + about how to interpret it. + </para> + + <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index"> + <title><structname>pg_stat_progress_create_index</structname> View</title> + <tgroup cols="3"> + <thead> + <row> + <entry>Column</entry> + <entry>Type</entry> + <entry>Description</entry> + </row> + </thead> + + <tbody> + <row> + <entry><structfield>pid</structfield></entry> + <entry><type>integer</type></entry> + <entry>Process ID of backend.</entry> + </row> + <row> + <entry><structfield>datid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>datname</structfield></entry> + <entry><type>name</type></entry> + <entry>Name of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>relid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the table on which the index is being created.</entry> + </row> + <row> + <entry><structfield>phase</structfield></entry> + <entry><type>text</type></entry> + <entry> + Current processing phase of index creation. See <xref linkend='create-index-phases'/>. + </entry> + </row> + <row> + <entry><structfield>lockers_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of lockers to wait for, when applicable. + </entry> + </row> + <row> + <entry><structfield>lockers_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of lockers already waited for. + </entry> + </row> + <row> + <entry><structfield>current_locked_pid</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Process ID of the locker currently being waited for. + </entry> + </row> + <row> + <entry><structfield>blocks_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of blocks to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>blocks_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of blocks already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of tuples to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of tuples already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>partitions_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned table, this column is set to + the total number of partitions on which the index is to be created. + </entry> + </row> + <row> + <entry><structfield>partitions_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned table, this column is set to + the number of partitions on which the index has been completed. + </entry> + </row> + </tbody> + </tgroup> + </table> + + <table id="create-index-phases"> + <title>CREATE INDEX phases</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Phase</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry><literal>initializing</literal></entry> + <entry> + <command>CREATE INDEX</command> is preparing to create the index. This + phase is expected to be very brief. + </entry> + </row> + <row> + <entry><literal>waiting for old snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>building index</literal></entry> + <entry> + The index is being built by the access method-specific code. In this phase, + access methods that support progress reporting fill in their own progress data, + and the subphase is indicated in this column. Typically, + <structname>blocks_total</structname> and <structname>blocks_done</structname> + will contain progress data, as well as potentially + <structname>tuples_total</structname> and <structname>tuples_done</structname>. + </entry> + </row> + <row> + <entry><literal>waiting for writer snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially write into the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>validating index scan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching + for tuples that need to be validated. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the index) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>sorting index scan results</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the + previous phase. + </entry> + </row> + <row> + <entry><literal>validating index heapscan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the table + to validate the index tuples collected in the previous two phases. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the table) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>waiting for reader snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. This + phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + </tbody> + </tgroup> + </table> + + </sect2> + <sect2 id="vacuum-progress-reporting"> <title>VACUUM Progress Reporting</title> diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index 6e96d24ca22..54273f754f6 100644 --- a/src/backend/access/brin/brin.c +++ b/src/backend/access/brin/brin.c @@ -112,6 +112,7 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = brincostestimate; amroutine->amoptions = brinoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = brinvalidate; amroutine->ambeginscan = brinbeginscan; amroutine->amrescan = brinrescan; @@ -719,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Now scan the relation. No syncscan allowed here because we want the * heap blocks in physical order. */ - reltuples = table_index_build_scan(heap, index, indexInfo, false, + reltuples = table_index_build_scan(heap, index, indexInfo, false, false, brinbuildCallback, (void *) state, NULL); /* process the final batch */ @@ -1236,7 +1237,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel, * cases. */ state->bs_currRangeStart = heapBlk; - table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, + table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, false, heapBlk, scanNumBlks, brinbuildCallback, (void *) state, NULL); diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c index b02f69b0dcb..b4b0213f76f 100644 --- a/src/backend/access/gin/gininsert.c +++ b/src/backend/access/gin/gininsert.c @@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Do the heap scan. We disallow sync scan here because dataPlaceToPage * prefers to receive tuples in TID order. */ - reltuples = table_index_build_scan(heap, index, indexInfo, false, + reltuples = table_index_build_scan(heap, index, indexInfo, false, false, ginBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c index afc20232ace..d2360eeafb0 100644 --- a/src/backend/access/gin/ginutil.c +++ b/src/backend/access/gin/ginutil.c @@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gincostestimate; amroutine->amoptions = ginoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = ginvalidate; amroutine->ambeginscan = ginbeginscan; amroutine->amrescan = ginrescan; diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 2fddb23496d..f44c922b5d6 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gistcostestimate; amroutine->amoptions = gistoptions; amroutine->amproperty = gistproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = gistvalidate; amroutine->ambeginscan = gistbeginscan; amroutine->amrescan = gistrescan; diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index 3652fde5bb1..771bd2962a7 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) /* * Do the heap scan. */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, gistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c index 5cc12a17130..6b38dd9c0e9 100644 --- a/src/backend/access/hash/hash.c +++ b/src/backend/access/hash/hash.c @@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = hashcostestimate; amroutine->amoptions = hashoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = hashvalidate; amroutine->ambeginscan = hashbeginscan; amroutine->amrescan = hashrescan; @@ -160,7 +161,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo) buildstate.heapRel = heap; /* do the heap scan */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, hashbuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index ce54b16f342..d22ef73555d 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -57,6 +57,8 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, OffsetNumber tupoffset); +static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); + static const TableAmRoutine heapam_methods; @@ -1109,6 +1111,7 @@ heapam_index_build_range_scan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -1129,6 +1132,7 @@ heapam_index_build_range_scan(Relation heapRelation, Snapshot snapshot; bool need_unregister_snapshot = false; TransactionId OldestXmin; + BlockNumber previous_blkno = InvalidBlockNumber; BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; @@ -1216,6 +1220,25 @@ heapam_index_build_range_scan(Relation heapRelation, hscan = (HeapScanDesc) scan; + /* Publish number of blocks to scan */ + if (progress) + { + BlockNumber nblocks; + + if (hscan->rs_base.rs_parallel != NULL) + { + ParallelBlockTableScanDesc pbscan; + + pbscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel; + nblocks = pbscan->phs_nblocks; + } + else + nblocks = hscan->rs_nblocks; + + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + nblocks); + } + /* * Must call GetOldestXmin() with SnapshotAny. Should never call * GetOldestXmin() with MVCC snapshot. (It's especially worth checking @@ -1248,6 +1271,19 @@ heapam_index_build_range_scan(Relation heapRelation, CHECK_FOR_INTERRUPTS(); + /* Report scan progress, if asked to. */ + if (progress) + { + BlockNumber blocks_done = heapam_scan_get_blocks_done(hscan); + + if (blocks_done != previous_blkno) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blocks_done); + previous_blkno = blocks_done; + } + } + /* * When dealing with a HOT-chain of updated tuples, we want to index * the values of the live tuple (if any), but index it under the TID @@ -1589,6 +1625,25 @@ heapam_index_build_range_scan(Relation heapRelation, } } + /* Report scan progress one last time. */ + if (progress) + { + BlockNumber blks_done; + + if (hscan->rs_base.rs_parallel != NULL) + { + ParallelBlockTableScanDesc pbscan; + + pbscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel; + blks_done = pbscan->phs_nblocks; + } + else + blks_done = hscan->rs_nblocks; + + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blks_done); + } + table_endscan(scan); /* we can now forget our snapshot, if set and registered by us */ @@ -1625,6 +1680,7 @@ heapam_index_validate_scan(Relation heapRelation, BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; bool in_index[MaxHeapTuplesPerPage]; + BlockNumber previous_blkno = InvalidBlockNumber; /* state variables for the merge */ ItemPointer indexcursor = NULL; @@ -1665,6 +1721,9 @@ heapam_index_validate_scan(Relation heapRelation, false); /* syncscan not OK */ hscan = (HeapScanDesc) scan; + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + hscan->rs_nblocks); + /* * Scan all tuples matching the snapshot. */ @@ -1678,6 +1737,14 @@ heapam_index_validate_scan(Relation heapRelation, state->htups += 1; + if ((previous_blkno == InvalidBlockNumber) || + (hscan->rs_cblock != previous_blkno)) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + hscan->rs_cblock); + previous_blkno = hscan->rs_cblock; + } + /* * As commented in table_index_build_scan, we should index heap-only * tuples under the TIDs of their root tuples; so when we advance onto @@ -1838,6 +1905,46 @@ heapam_index_validate_scan(Relation heapRelation, indexInfo->ii_PredicateState = NULL; } +/* + * Return the number of blocks that have been read by this scan since + * starting. This is meant for progress reporting rather than be fully + * accurate: in a parallel scan, workers can be concurrently reading blocks + * further ahead than what we report. + */ +static BlockNumber +heapam_scan_get_blocks_done(HeapScanDesc hscan) +{ + ParallelBlockTableScanDesc bpscan = NULL; + BlockNumber startblock; + BlockNumber blocks_done; + + if (hscan->rs_base.rs_parallel != NULL) + { + bpscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel; + startblock = bpscan->phs_startblock; + } + else + startblock = hscan->rs_startblock; + + /* + * Might have wrapped around the end of the relation, if startblock was + * not zero. + */ + if (hscan->rs_cblock > startblock) + blocks_done = hscan->rs_cblock - startblock; + else + { + BlockNumber nblocks; + + nblocks = bpscan != NULL ? bpscan->phs_nblocks : hscan->rs_nblocks; + blocks_done = nblocks - startblock + + hscan->rs_cblock; + } + + return blocks_done; +} + + /* ------------------------------------------------------------------------ * Planner related callbacks for the heap AM diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index ac6f1eb3423..7370379c6a1 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -22,6 +22,7 @@ #include "access/nbtxlog.h" #include "access/relscan.h" #include "access/xlog.h" +#include "commands/progress.h" #include "commands/vacuum.h" #include "miscadmin.h" #include "nodes/execnodes.h" @@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = btcostestimate; amroutine->amoptions = btoptions; amroutine->amproperty = btproperty; + amroutine->ambuildphasename = btbuildphasename; amroutine->amvalidate = btvalidate; amroutine->ambeginscan = btbeginscan; amroutine->amrescan = btrescan; @@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (needLock) UnlockRelationForExtension(rel, ExclusiveLock); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + num_pages); + /* Quit if we've scanned the whole relation */ if (blkno >= num_pages) break; @@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, for (; blkno < num_pages; blkno++) { btvacuumpage(&vstate, blkno, blkno); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blkno); } } diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index 14d95457682..b333a7f82b2 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -66,6 +66,7 @@ #include "access/xlog.h" #include "access/xloginsert.h" #include "catalog/index.h" +#include "commands/progress.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/smgr.h" @@ -298,7 +299,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate, static void _bt_leader_participate_as_worker(BTBuildState *buildstate); static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem); + Sharedsort *sharedsort2, int sortmem, + bool progress); /* @@ -394,6 +396,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Save as primary spool */ buildstate->spool = btspool; + /* Report heap scan phase started */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN); + /* Attempt to launch parallel worker scan when required */ if (indexInfo->ii_ParallelWorkers > 0) _bt_begin_parallel(buildstate, indexInfo->ii_Concurrent, @@ -480,13 +486,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Fill spool using either serial or parallel heap scan */ if (!buildstate->btleader) - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, true, _bt_build_callback, (void *) buildstate, NULL); else reltuples = _bt_parallel_heapscan(buildstate, &indexInfo->ii_BrokenHotChain); + /* + * Set the progress target for the next phase. Reset the block number + * values set by table_index_build_scan + */ + { + const int index[] = { + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE + }; + const int64 val[] = { + buildstate->indtuples, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } + /* okay, all heap tuples are spooled */ if (buildstate->spool2 && !buildstate->havedead) { @@ -535,9 +559,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) } #endif /* BTREE_BUILD_STATS */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_1); tuplesort_performsort(btspool->sortstate); if (btspool2) + { + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_2); tuplesort_performsort(btspool2->sortstate); + } wstate.heap = btspool->heap; wstate.index = btspool->index; @@ -554,6 +584,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) wstate.btws_pages_written = 0; wstate.btws_zeropage = NULL; /* until needed */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_LEAF_LOAD); _bt_load(&wstate, btspool, btspool2); } @@ -1098,6 +1130,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) int i, keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index); SortSupport sortKeys; + long tuples_done = 0L; if (merge) { @@ -1202,6 +1235,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) _bt_buildadd(wstate, state, itup2); itup2 = tuplesort_getindextuple(btspool2->sortstate, true); } + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } pfree(sortKeys); } @@ -1216,6 +1253,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) state = _bt_pagestate(wstate, 0); _bt_buildadd(wstate, state, itup); + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } } @@ -1528,7 +1569,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate) /* Perform work common to all participants */ _bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared, btleader->sharedsort, btleader->sharedsort2, - sortmem); + sortmem, true); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1619,7 +1660,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) /* Perform sorting of spool, and possibly a spool2 */ sortmem = maintenance_work_mem / btshared->scantuplesortstates; _bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort, - sharedsort2, sortmem); + sharedsort2, sortmem, false); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1648,7 +1689,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem) + Sharedsort *sharedsort2, int sortmem, bool progress) { SortCoordinate coordinate; BTBuildState buildstate; @@ -1705,9 +1746,10 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, /* Join parallel scan */ indexInfo = BuildIndexInfo(btspool->index); indexInfo->ii_Concurrent = btshared->isconcurrent; - scan = table_beginscan_parallel(btspool->heap, ParallelTableScanFromBTShared(btshared)); + scan = table_beginscan_parallel(btspool->heap, + ParallelTableScanFromBTShared(btshared)); reltuples = table_index_build_scan(btspool->heap, btspool->index, indexInfo, - true, _bt_build_callback, + true, progress, _bt_build_callback, (void *) &buildstate, scan); /* diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c index 140ac920265..a0cc9dba6d9 100644 --- a/src/backend/access/nbtree/nbtutils.c +++ b/src/backend/access/nbtree/nbtutils.c @@ -20,6 +20,7 @@ #include "access/nbtree.h" #include "access/reloptions.h" #include "access/relscan.h" +#include "commands/progress.h" #include "miscadmin.h" #include "utils/array.h" #include "utils/datum.h" @@ -2051,6 +2052,29 @@ btproperty(Oid index_oid, int attno, } } +/* + * btbuildphasename() -- Return name of index build phase. + */ +char * +btbuildphasename(int64 phasenum) +{ + switch (phasenum) + { + case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE: + return "initializing"; + case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN: + return "scanning table"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_1: + return "sorting live tuples"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_2: + return "sorting dead tuples"; + case PROGRESS_BTREE_PHASE_LEAF_LOAD: + return "loading tuples in tree"; + default: + return NULL; + } +} + /* * _bt_truncate() -- create tuple without unneeded suffix attributes. * diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c index 390ad9ac51f..282d6998cf0 100644 --- a/src/backend/access/spgist/spginsert.c +++ b/src/backend/access/spgist/spginsert.c @@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo) "SP-GiST build temporary context", ALLOCSET_DEFAULT_SIZES); - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, spgistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c index 8e63c1fad25..45472db147b 100644 --- a/src/backend/access/spgist/spgutils.c +++ b/src/backend/access/spgist/spgutils.c @@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = spgcostestimate; amroutine->amoptions = spgoptions; amroutine->amproperty = spgproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = spgvalidate; amroutine->ambeginscan = spgbeginscan; amroutine->amrescan = spgrescan; diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 0d9d405c548..6c6818e891c 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -51,9 +51,9 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" -#include "commands/tablecmds.h" #include "commands/event_trigger.h" #include "commands/progress.h" +#include "commands/tablecmds.h" #include "commands/trigger.h" #include "executor/executor.h" #include "miscadmin.h" @@ -2047,7 +2047,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode) * to acquire an exclusive lock on our table. The lock code will * detect deadlock and error out properly. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* Finish invalidation of index and mark it as dead */ index_concurrently_set_dead(heapId, indexId); @@ -2063,7 +2063,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode) * Wait till every transaction that saw the old index state has * finished. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * Re-open relations to allow us to complete our actions. @@ -2712,6 +2712,25 @@ index_build(Relation heapRelation, save_sec_context | SECURITY_RESTRICTED_OPERATION); save_nestlevel = NewGUCNestLevel(); + /* Set up initial progress report status */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_BUILD, + PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE, + 0, 0, 0, 0 + }; + + pgstat_progress_update_multi_param(6, index, val); + } + /* * Call the access method's build procedure */ @@ -3000,6 +3019,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) int save_sec_context; int save_nestlevel; + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN, + 0, 0, 0, 0 + }; + pgstat_progress_update_multi_param(5, index, val); + } + /* Open and lock the parent heap relation */ heapRelation = table_open(heapId, ShareUpdateExclusiveLock); /* And the target index relation */ @@ -3030,6 +3064,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) */ ivinfo.index = indexRelation; ivinfo.analyze_only = false; + ivinfo.report_progress = true; /* XXX only for btree? */ ivinfo.estimated_count = true; ivinfo.message_level = DEBUG2; ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples; @@ -3047,15 +3082,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) NULL, false); state.htups = state.itups = state.tups_inserted = 0; + /* ambulkdelete updates progress metrics */ (void) index_bulk_delete(&ivinfo, NULL, validate_index_callback, (void *) &state); /* Execute the sort */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } tuplesort_performsort(state.tuplesort); /* - * Now scan the heap and "merge" it with the index + * Now scan the heap and "merge" it with the index. */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE + }; + + pgstat_progress_update_multi_param(1, index, val); + } table_index_validate_scan(heapRelation, indexRelation, indexInfo, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index b89df70653e..acf277473c1 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -934,6 +934,33 @@ CREATE VIEW pg_stat_progress_cluster AS FROM pg_stat_get_progress_info('CLUSTER') AS S LEFT JOIN pg_database D ON S.datid = D.oid; +CREATE VIEW pg_stat_progress_create_index AS + SELECT + S.pid AS pid, S.datid AS datid, D.datname AS datname, + S.relid AS relid, + CASE S.param10 WHEN 0 THEN 'initializing' + WHEN 1 THEN 'waiting for old snapshots' + WHEN 2 THEN 'building index' || + COALESCE((': ' || pg_indexam_progress_phasename(S.param9::oid, S.param11)), + '') + WHEN 3 THEN 'waiting for writer snapshots' + WHEN 4 THEN 'index validation: scan index' + WHEN 5 THEN 'index validation: sort index scan results' + WHEN 6 THEN 'index validation: scan heap' + WHEN 7 THEN 'waiting for reader snapshots' + END as phase, + S.param4 AS lockers_total, + S.param5 AS lockers_done, + S.param6 AS current_locker_pid, + S.param16 AS blocks_total, + S.param17 AS blocks_done, + S.param12 AS tuples_total, + S.param13 AS tuples_done, + S.param14 AS partitions_total, + S.param15 AS partitions_done + FROM pg_stat_get_progress_info('CREATE INDEX') AS S + LEFT JOIN pg_database D ON S.datid = D.oid; + CREATE VIEW pg_user_mappings AS SELECT U.oid AS umid, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 53971fc7258..348d5432977 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -36,6 +36,7 @@ #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" +#include "commands/progress.h" #include "commands/tablecmds.h" #include "commands/tablespace.h" #include "mb/pg_wchar.h" @@ -47,10 +48,12 @@ #include "parser/parse_func.h" #include "parser/parse_oper.h" #include "partitioning/partdesc.h" +#include "pgstat.h" #include "rewrite/rewriteManip.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/acl.h" #include "utils/builtins.h" #include "utils/fmgroids.h" @@ -334,7 +337,7 @@ CheckIndexCompatible(Oid oldId, * doesn't show up in the output, we know we can forget about it. */ static void -WaitForOlderSnapshots(TransactionId limitXmin) +WaitForOlderSnapshots(TransactionId limitXmin, bool progress) { int n_old_snapshots; int i; @@ -343,6 +346,8 @@ WaitForOlderSnapshots(TransactionId limitXmin) old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false, PROC_IS_AUTOVACUUM | PROC_IN_VACUUM, &n_old_snapshots); + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots); for (i = 0; i < n_old_snapshots; i++) { @@ -378,7 +383,19 @@ WaitForOlderSnapshots(TransactionId limitXmin) } if (VirtualTransactionIdIsValid(old_snapshots[i])) + { + if (progress) + { + PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId); + + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); + } VirtualXactLock(old_snapshots[i], true); + } + + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1); } } @@ -452,6 +469,15 @@ DefineIndex(Oid relationId, Snapshot snapshot; int i; + + /* + * Start progress report. If we're building a partition, this was already + * done. + */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX, + relationId); + /* * count key attributes in index */ @@ -668,6 +694,9 @@ DefineIndex(Oid relationId, accessMethodId = accessMethodForm->oid; amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler); + pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID, + accessMethodId); + if (stmt->unique && !amRoutine->amcanunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -948,6 +977,11 @@ DefineIndex(Oid relationId, if (!OidIsValid(indexRelationId)) { table_close(rel, NoLock); + + /* If this is the top-level index, we're done */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -973,6 +1007,9 @@ DefineIndex(Oid relationId, TupleDesc parentDesc; Oid *opfamOids; + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL, + nparts); + memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts); parentDesc = CreateTupleDescCopy(RelationGetDescr(rel)); @@ -1122,6 +1159,8 @@ DefineIndex(Oid relationId, skip_build, quiet); } + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, + i + 1); pfree(attmap); } @@ -1156,6 +1195,8 @@ DefineIndex(Oid relationId, * Indexes on partitioned tables are not themselves built, so we're * done here. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); return address; } @@ -1163,6 +1204,11 @@ DefineIndex(Oid relationId, { /* Close the heap and we're done, in the non-concurrent case */ table_close(rel, NoLock); + + /* If this is the top-level index, we're done. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -1214,7 +1260,9 @@ DefineIndex(Oid relationId, * exclusive lock on our table. The lock code will detect deadlock and * error out properly. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_1); + WaitForLockers(heaplocktag, ShareLock, true); /* * At this moment we are sure that there are no transactions with the @@ -1255,7 +1303,9 @@ DefineIndex(Oid relationId, * We once again wait until no transaction can have the table open with * the index marked as read-only for updates. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_2); + WaitForLockers(heaplocktag, ShareLock, true); /* * Now take the "reference snapshot" that will be used by validate_index() @@ -1312,7 +1362,9 @@ DefineIndex(Oid relationId, * before the reference snap was taken, we have to wait out any * transactions that might have older snapshots. */ - WaitForOlderSnapshots(limitXmin); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_3); + WaitForOlderSnapshots(limitXmin, true); /* * Index can now be marked valid -- update its pg_index entry @@ -1334,6 +1386,8 @@ DefineIndex(Oid relationId, */ UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock); + pgstat_progress_end_command(); + return address; } @@ -2913,7 +2967,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * DefineIndex() for more details. */ - WaitForLockersMultiple(lockTags, ShareLock); + WaitForLockersMultiple(lockTags, ShareLock, false); CommitTransactionCommand(); forboth(lc, indexIds, lc2, newIndexIds) @@ -2955,7 +3009,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * for more details. */ - WaitForLockersMultiple(lockTags, ShareLock); + WaitForLockersMultiple(lockTags, ShareLock, false); CommitTransactionCommand(); foreach(lc, newIndexIds) @@ -3003,7 +3057,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * before the reference snap was taken, we have to wait out any * transactions that might have older snapshots. */ - WaitForOlderSnapshots(limitXmin); + WaitForOlderSnapshots(limitXmin, false); CommitTransactionCommand(); } @@ -3074,7 +3128,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * index_drop() for more details. */ - WaitForLockersMultiple(lockTags, AccessExclusiveLock); + WaitForLockersMultiple(lockTags, AccessExclusiveLock, false); foreach(lc, indexIds) { @@ -3096,7 +3150,7 @@ ReindexRelationConcurrently(Oid relationOid, int options) * Drop the old indexes. */ - WaitForLockersMultiple(lockTags, AccessExclusiveLock); + WaitForLockersMultiple(lockTags, AccessExclusiveLock, false); PushActiveSnapshot(GetTransactionSnapshot()); diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index cd56dca3aef..215f1463bb1 100644 --- a/src/backend/storage/ipc/standby.c +++ b/src/backend/storage/ipc/standby.c @@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag) */ VirtualTransactionId *backends; - backends = GetLockConflicts(&locktag, AccessExclusiveLock); + backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL); ResolveRecoveryConflictWithVirtualXIDs(backends, PROCSIG_RECOVERY_CONFLICT_LOCK); } diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c index e688ba81170..0b04b093782 100644 --- a/src/backend/storage/lmgr/lmgr.c +++ b/src/backend/storage/lmgr/lmgr.c @@ -19,9 +19,12 @@ #include "access/transam.h" #include "access/xact.h" #include "catalog/catalog.h" +#include "commands/progress.h" #include "miscadmin.h" +#include "pgstat.h" #include "storage/lmgr.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/inval.h" @@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg) * after we obtained our initial list of lockers, we will not wait for them. */ void -WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) +WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress) { List *holders = NIL; ListCell *lc; + int total = 0; + int done = 0; /* Done if no locks to wait for */ if (list_length(locktags) == 0) @@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) foreach(lc, locktags) { LOCKTAG *locktag = lfirst(lc); + int count; - holders = lappend(holders, GetLockConflicts(locktag, lockmode)); + holders = lappend(holders, + GetLockConflicts(locktag, lockmode, + progress ? &count : NULL)); + total += count; } + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total); + /* * Note: GetLockConflicts() never reports our own xid, hence we need not * check for that. Also, prepared xacts are not reported, which is fine @@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) while (VirtualTransactionIdIsValid(*lockholders)) { + /* + * If requested, publish who we're going to wait for. This is not + * 100% accurate if they're already gone, but we don't care. + */ + if (progress) + { + PGPROC *holder = BackendIdGetProc(lockholders->backendId); + + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); + } VirtualXactLock(*lockholders, true); lockholders++; + + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done); } } + if (progress) + { + const int index[] = { + PROGRESS_WAITFOR_TOTAL, + PROGRESS_WAITFOR_DONE, + PROGRESS_WAITFOR_CURRENT_PID + }; + const int64 values[] = { + 0, 0, 0 + }; + pgstat_progress_update_multi_param(3, index, values); + } list_free_deep(holders); } @@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) * Same as WaitForLockersMultiple, for a single lock tag. */ void -WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode) +WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress) { List *l; l = list_make1(&heaplocktag); - WaitForLockersMultiple(l, lockmode); + WaitForLockersMultiple(l, lockmode, progress); list_free(l); } diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c index 78fdbd6ff88..c8958766f1e 100644 --- a/src/backend/storage/lmgr/lock.c +++ b/src/backend/storage/lmgr/lock.c @@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * xacts merely awaiting such a lock are NOT reported. * * The result array is palloc'd and is terminated with an invalid VXID. + * *countp, if not null, is updated to the number of items set. * * Of course, the result could be out of date by the time it's returned, * so use of this function has to be thought about carefully. @@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * uses of the result. */ VirtualTransactionId * -GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) +GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp) { static VirtualTransactionId *vxids; LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid; @@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) LWLockRelease(partitionLock); vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } @@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c index 060ffe501ec..e81d6cc0562 100644 --- a/src/backend/utils/adt/amutils.c +++ b/src/backend/utils/adt/amutils.c @@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS) return indexam_property(fcinfo, propname, InvalidOid, relid, attno); } + +/* + * Return the name of the given phase, as used for progress reporting by the + * given AM. + */ +Datum +pg_indexam_progress_phasename(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + int32 phasenum = PG_GETARG_INT32(1); + IndexAmRoutine *routine; + char *name; + + routine = GetIndexAmRoutineByAmId(amoid, true); + if (routine == NULL || !routine->ambuildphasename) + PG_RETURN_NULL(); + + name = routine->ambuildphasename(phasenum); + if (!name) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(CStringGetTextDatum(name)); +} diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 90a817a25c5..7c2afe64272 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -470,6 +470,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS) cmdtype = PROGRESS_COMMAND_VACUUM; else if (pg_strcasecmp(cmd, "CLUSTER") == 0) cmdtype = PROGRESS_COMMAND_CLUSTER; + else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0) + cmdtype = PROGRESS_COMMAND_CREATE_INDEX; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index 653ddc976ba..09a7404267c 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +/* name of phase as used in progress reporting */ +typedef char *(*ambuildphasename_function) (int64 phasenum); + /* validate definition of an opclass for this AM */ typedef bool (*amvalidate_function) (Oid opclassoid); @@ -213,6 +216,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 70c7351a08c..9717183ef23 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo { Relation index; /* the index being vacuumed */ bool analyze_only; /* ANALYZE (without any actual vacuum) */ + bool report_progress; /* emit progress.h status reports */ bool estimated_count; /* num_heap_tuples is an estimate */ int message_level; /* ereport level for progress messages */ double num_heap_tuples; /* tuples remaining in heap */ diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h index a1ffa983365..8a727872dae 100644 --- a/src/include/access/nbtree.h +++ b/src/include/access/nbtree.h @@ -671,6 +671,16 @@ typedef BTScanOpaqueData *BTScanOpaque; #define SK_BT_DESC (INDOPTION_DESC << SK_BT_INDOPTION_SHIFT) #define SK_BT_NULLS_FIRST (INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT) +/* + * Constant definition for progress reporting. Phase numbers must match + * btbuildphasename. + */ +/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */ +#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN 2 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_1 3 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_2 4 +#define PROGRESS_BTREE_PHASE_LEAF_LOAD 5 + /* * external entry points for btree, in nbtree.c */ @@ -784,6 +794,7 @@ extern bytea *btoptions(Datum reloptions, bool validate); extern bool btproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +extern char *btbuildphasename(int64 phasenum); extern IndexTuple _bt_truncate(Relation rel, IndexTuple lastleft, IndexTuple firstright, BTScanInsert itup_key); extern int _bt_keep_natts_fast(Relation rel, IndexTuple lastleft, diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 9bc1d24b4a6..3ff2918a42b 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -492,6 +492,7 @@ typedef struct TableAmRoutine struct IndexInfo *index_nfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber end_blockno, IndexBuildCallback callback, @@ -1339,6 +1340,11 @@ table_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * so here because the AM might reject some of the tuples for its own reasons, * such as being unable to store NULLs. * + * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go + * along. Also, if that flag is true and a scan descriptor is not passed (ie. + * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is + * updated at the beginning. For parallel scans, caller is expected to have + * set the total number of blocks prior to calling this function. * * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect * any potentially broken HOT chains. Currently, we set this if there are any @@ -1352,6 +1358,7 @@ table_index_build_scan(Relation heap_rel, Relation index_rel, struct IndexInfo *index_nfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, TableScanDesc scan) @@ -1361,6 +1368,7 @@ table_index_build_scan(Relation heap_rel, index_nfo, allow_sync, false, + progress, 0, InvalidBlockNumber, callback, @@ -1384,6 +1392,7 @@ table_index_build_range_scan(Relation heap_rel, struct IndexInfo *index_nfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -1395,6 +1404,7 @@ table_index_build_range_scan(Relation heap_rel, index_nfo, allow_sync, anyvisible, + progress, start_blockno, numblocks, callback, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index eac909109c5..a7050edca09 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -924,6 +924,10 @@ proname => 'pg_index_column_has_property', provolatile => 's', prorettype => 'bool', proargtypes => 'regclass int4 text', prosrc => 'pg_index_column_has_property' }, +{ oid => '676', descr => 'return name of given index build phase', + proname => 'pg_indexam_progress_phasename', provolatile => 'i', + prorettype => 'text', proargtypes => 'oid int8', + prosrc => 'pg_indexam_progress_phasename' }, { oid => '339', proname => 'poly_same', prorettype => 'bool', @@ -5122,9 +5126,9 @@ proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't', provolatile => 's', proparallel => 'r', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', - proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}', + proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12,param13,param14,param15,param16,param17,param18,param19,param20}', prosrc => 'pg_stat_get_progress_info' }, { oid => '3099', descr => 'statistics: information about currently active replication', diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h index 04542d9e923..079303cff46 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -44,7 +44,7 @@ #define PROGRESS_CLUSTER_HEAP_BLKS_SCANNED 6 #define PROGRESS_CLUSTER_INDEX_REBUILD_COUNT 7 -/* Phases of cluster (as dvertised via PROGRESS_CLUSTER_PHASE) */ +/* Phases of cluster (as advertised via PROGRESS_CLUSTER_PHASE) */ #define PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP 1 #define PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP 2 #define PROGRESS_CLUSTER_PHASE_SORT_TUPLES 3 @@ -57,4 +57,39 @@ #define PROGRESS_CLUSTER_COMMAND_CLUSTER 1 #define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL 2 +/* Progress parameters for CREATE INDEX */ +/* 3, 4 and 5 reserved for "waitfor" metrics */ +#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID 8 +#define PROGRESS_CREATEIDX_PHASE 9 /* AM-agnostic phase # */ +#define PROGRESS_CREATEIDX_SUBPHASE 10 /* phase # filled by AM */ +#define PROGRESS_CREATEIDX_TUPLES_TOTAL 11 +#define PROGRESS_CREATEIDX_TUPLES_DONE 12 +#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL 13 +#define PROGRESS_CREATEIDX_PARTITIONS_DONE 14 +/* 15 and 16 reserved for "block number" metrics */ + +/* Phases of CREATE INDEX (as advertised via PROGRESS_CREATEIDX_PHASE) */ +#define PROGRESS_CREATEIDX_PHASE_WAIT_1 1 +#define PROGRESS_CREATEIDX_PHASE_BUILD 2 +#define PROGRESS_CREATEIDX_PHASE_WAIT_2 3 +#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN 4 +#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN 5 +#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE 6 +#define PROGRESS_CREATEIDX_PHASE_WAIT_3 7 + +/* + * Subphases of CREATE INDEX, for index_build. + */ +#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE 1 +/* Additional phases are defined by each AM */ + +/* Lock holder wait counts */ +#define PROGRESS_WAITFOR_TOTAL 3 +#define PROGRESS_WAITFOR_DONE 4 +#define PROGRESS_WAITFOR_CURRENT_PID 5 + +/* Block numbers in a generic relation scan */ +#define PROGRESS_SCAN_BLOCKS_TOTAL 15 +#define PROGRESS_SCAN_BLOCKS_DONE 16 + #endif diff --git a/src/include/pgstat.h b/src/include/pgstat.h index c080fa6388f..53d4a9c4319 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -951,10 +951,11 @@ typedef enum ProgressCommandType { PROGRESS_COMMAND_INVALID, PROGRESS_COMMAND_VACUUM, - PROGRESS_COMMAND_CLUSTER + PROGRESS_COMMAND_CLUSTER, + PROGRESS_COMMAND_CREATE_INDEX } ProgressCommandType; -#define PGSTAT_NUM_PROGRESS_PARAM 10 +#define PGSTAT_NUM_PROGRESS_PARAM 20 /* ---------- * Shared-memory data structures diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h index 3d705faba5c..4f2872de35f 100644 --- a/src/include/storage/lmgr.h +++ b/src/include/storage/lmgr.h @@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel, extern bool ConditionalXactLockTableWait(TransactionId xid); /* Lock VXIDs, specified by conflicting locktags */ -extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode); -extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode); +extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress); +extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress); /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */ extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid); diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h index badf7fd682b..048947c50d4 100644 --- a/src/include/storage/lock.h +++ b/src/include/storage/lock.h @@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode); extern bool LockHasWaiters(const LOCKTAG *locktag, LOCKMODE lockmode, bool sessionLock); extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag, - LOCKMODE lockmode); + LOCKMODE lockmode, int *countp); extern void AtPrepare_Locks(void); extern void PostPrepare_Locks(TransactionId xid); extern int LockCheckConflicts(LockMethod lockMethodTable, diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index d5f309fbfbe..51d7a150cfb 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1856,7 +1856,33 @@ pg_stat_progress_cluster| SELECT s.pid, s.param6 AS heap_blks_total, s.param7 AS heap_blks_scanned, s.param8 AS index_rebuild_count - FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20) + LEFT JOIN pg_database d ON ((s.datid = d.oid))); +pg_stat_progress_create_index| SELECT s.pid, + s.datid, + d.datname, + s.relid, + CASE s.param10 + WHEN 0 THEN 'initializing'::text + WHEN 1 THEN 'waiting for old snapshots'::text + WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param9)::oid, s.param11)), ''::text)) + WHEN 3 THEN 'waiting for writer snapshots'::text + WHEN 4 THEN 'index validation: scan index'::text + WHEN 5 THEN 'index validation: sort index scan results'::text + WHEN 6 THEN 'index validation: scan heap'::text + WHEN 7 THEN 'waiting for reader snapshots'::text + ELSE NULL::text + END AS phase, + s.param4 AS lockers_total, + s.param5 AS lockers_done, + s.param6 AS current_locker_pid, + s.param16 AS blocks_total, + s.param17 AS blocks_done, + s.param12 AS tuples_total, + s.param13 AS tuples_done, + s.param14 AS partitions_total, + s.param15 AS partitions_done + FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20) LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_progress_vacuum| SELECT s.pid, s.datid, @@ -1878,7 +1904,7 @@ pg_stat_progress_vacuum| SELECT s.pid, s.param5 AS index_vacuum_count, s.param6 AS max_dead_tuples, s.param7 AS num_dead_tuples - FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20) LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_replication| SELECT s.pid, s.usesysid, -- 2.17.1 --UlVJffcvxoiEqYs2 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0002-report-partial-progress-of-other-index-AMs.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v7 1/2] Report progress of CREATE INDEX operations @ 2019-03-25 16:29 Alvaro Herrera <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Alvaro Herrera @ 2019-03-25 16:29 UTC (permalink / raw) --- contrib/amcheck/verify_nbtree.c | 2 +- contrib/bloom/blinsert.c | 2 +- contrib/bloom/blutils.c | 1 + doc/src/sgml/indexam.sgml | 13 ++ doc/src/sgml/monitoring.sgml | 224 ++++++++++++++++++++++- src/backend/access/brin/brin.c | 5 +- src/backend/access/gin/gininsert.c | 2 +- src/backend/access/gin/ginutil.c | 1 + src/backend/access/gist/gist.c | 1 + src/backend/access/gist/gistbuild.c | 2 +- src/backend/access/hash/hash.c | 3 +- src/backend/access/heap/heapam_handler.c | 75 ++++++++ src/backend/access/nbtree/nbtree.c | 9 + src/backend/access/nbtree/nbtsort.c | 61 +++++- src/backend/access/nbtree/nbtutils.c | 24 +++ src/backend/access/spgist/spginsert.c | 2 +- src/backend/access/spgist/spgutils.c | 1 + src/backend/catalog/index.c | 67 ++++++- src/backend/catalog/system_views.sql | 27 +++ src/backend/commands/indexcmds.c | 52 +++++- src/backend/storage/ipc/standby.c | 2 +- src/backend/storage/lmgr/lmgr.c | 46 ++++- src/backend/storage/lmgr/lock.c | 7 +- src/backend/utils/adt/amutils.c | 23 +++ src/backend/utils/adt/pgstatfuncs.c | 2 + src/include/access/amapi.h | 4 + src/include/access/genam.h | 1 + src/include/access/nbtree.h | 11 ++ src/include/access/tableam.h | 10 + src/include/catalog/pg_proc.dat | 10 +- src/include/commands/progress.h | 35 ++++ src/include/pgstat.h | 5 +- src/include/storage/lmgr.h | 4 +- src/include/storage/lock.h | 2 +- src/test/regress/expected/rules.out | 30 ++- 35 files changed, 728 insertions(+), 38 deletions(-) diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c index 9ecb1999e34..b55178328c8 100644 --- a/contrib/amcheck/verify_nbtree.c +++ b/contrib/amcheck/verify_nbtree.c @@ -566,7 +566,7 @@ bt_check_every_level(Relation rel, Relation heaprel, bool heapkeyspace, RelationGetRelationName(state->rel), RelationGetRelationName(state->heaprel)); - table_index_build_scan(state->heaprel, state->rel, indexinfo, true, + table_index_build_scan(state->heaprel, state->rel, indexinfo, true, false, bt_tuple_present_callback, (void *) state, scan); ereport(DEBUG1, diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c index 1b8df7e1e84..48f35d39990 100644 --- a/contrib/bloom/blinsert.c +++ b/contrib/bloom/blinsert.c @@ -142,7 +142,7 @@ blbuild(Relation heap, Relation index, IndexInfo *indexInfo) initCachedPage(&buildstate); /* Do the heap scan */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, bloomBuildCallback, (void *) &buildstate, NULL); diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c index d078dfbd469..ee3bd562748 100644 --- a/contrib/bloom/blutils.c +++ b/contrib/bloom/blutils.c @@ -132,6 +132,7 @@ blhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = blcostestimate; amroutine->amoptions = bloptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = blvalidate; amroutine->ambeginscan = blbeginscan; amroutine->amrescan = blrescan; diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml index b56d3b3daa1..ff8290da9ff 100644 --- a/doc/src/sgml/indexam.sgml +++ b/doc/src/sgml/indexam.sgml @@ -127,6 +127,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; @@ -468,6 +469,18 @@ amproperty (Oid index_oid, int attno, <para> <programlisting> +char * +ambuildphasename (int64 phasenum); +</programlisting> + Return the textual name of the given build phase number. + The phase numbers are those reported during an index build via the + <function>pgstat_progress_update_param</function> interface. + The phase names are then exposed in the + <structname>pg_stat_progress_create_index</structname> view. + </para> + + <para> +<programlisting> bool amvalidate (Oid opclassoid); </programlisting> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index f1df14bdea8..8f5bbbb79e5 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -336,6 +336,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </entry> </row> + <row> + <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry> + <entry>One row for each backend running <command>CREATE INDEX</command>, showing + current progress. + See <xref linkend='create-index-progress-reporting'/>. + </entry> + </row> + <row> <entry><structname>pg_stat_progress_vacuum</structname><indexterm><primary>pg_stat_progress_vacuum</primary></indexterm></entry> <entry>One row for each backend (including autovacuum worker processes) running @@ -3403,10 +3411,224 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <para> <productname>PostgreSQL</productname> has the ability to report the progress of certain commands during command execution. Currently, the only commands - which support progress reporting are <command>VACUUM</command> and + which support progress reporting are <command>CREATE INDEX</command>, + <command>VACUUM</command> and <command>CLUSTER</command>. This may be expanded in the future. </para> + <sect2 id="create-index-progress-reporting"> + <title>CREATE INDEX Progress Reporting</title> + + <para> + Whenever <command>CREATE INDEX</command> is running, the + <structname>pg_stat_progress_create_index</structname> view will contain + one row for each backend that is currently creating indexes. The tables + below describe the information that will be reported and provide information + about how to interpret it. + </para> + + <table id="pg-stat-progress-create-index-view" xreflabel="pg_stat_progress_create_index"> + <title><structname>pg_stat_progress_create_index</structname> View</title> + <tgroup cols="3"> + <thead> + <row> + <entry>Column</entry> + <entry>Type</entry> + <entry>Description</entry> + </row> + </thead> + + <tbody> + <row> + <entry><structfield>pid</structfield></entry> + <entry><type>integer</type></entry> + <entry>Process ID of backend.</entry> + </row> + <row> + <entry><structfield>datid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>datname</structfield></entry> + <entry><type>name</type></entry> + <entry>Name of the database to which this backend is connected.</entry> + </row> + <row> + <entry><structfield>relid</structfield></entry> + <entry><type>oid</type></entry> + <entry>OID of the table on which the index is being created.</entry> + </row> + <row> + <entry><structfield>phase</structfield></entry> + <entry><type>text</type></entry> + <entry> + Current processing phase of index creation. See <xref linkend='create-index-phases'/>. + </entry> + </row> + <row> + <entry><structfield>lockers_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of lockers to wait for, when applicable. + </entry> + </row> + <row> + <entry><structfield>lockers_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of lockers already waited for. + </entry> + </row> + <row> + <entry><structfield>current_locked_pid</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Process ID of the locker currently being waited for. + </entry> + </row> + <row> + <entry><structfield>blocks_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of blocks to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>blocks_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of blocks already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Total number of tuples to be processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>tuples_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + Number of tuples already processed in the current phase. + </entry> + </row> + <row> + <entry><structfield>partitions_total</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + total number of partitions on which the index is to be created. + </entry> + </row> + <row> + <entry><structfield>partitions_done</structfield></entry> + <entry><type>bigint</type></entry> + <entry> + When creating an index on a partitioned, this column is set to the + number of partitions on which the index has been completed. + </entry> + </row> + </tbody> + </tgroup> + </table> + + <table id="create-index-phases"> + <title>CREATE INDEX phases</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Phase</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry><literal>initializing</literal></entry> + <entry> + <command>CREATE INDEX</command> is preparing to create the index. This + phase is expected to be very brief. + </entry> + </row> + <row> + <entry><literal>waiting for old snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>building index</literal></entry> + <entry> + The index is being built by the access method-specific code. In this phase, + access methods that support progress reporting fill in their own progress data, + and the subphase is indicated in this column. Typically, + <structname>blocks_total</structname> and <structname>blocks_done</structname> + will contain progress data, as well as potentially + <structname>tuples_total</structname> and <structname>tuples_done</structname>. + </entry> + </row> + <row> + <entry><literal>waiting for writer snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially write into the table to release their snapshots. + This phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + <row> + <entry><literal>validating index scan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the index searching + for tuples that need to be validated. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the index) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>sorting index scan results</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is sorting the output of the + previous phase. + </entry> + </row> + <row> + <entry><literal>validating index heapscan</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is scanning the table + to validate the index tuples collected in the previous two phases. + This phase is skipped when not in concurrent mode. + Columns <structname>blocks_total</structname> (set to the total size of the table) + and <structname>blocks_done</structname> contain the progress information for this phase. + </entry> + </row> + <row> + <entry><literal>waiting for reader snapshots</literal></entry> + <entry> + <command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions + that can potentially see the table to release their snapshots. This + phase is skipped when not in concurrent mode. + Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> + and <structname>current_locker_pid</structname> contain the progress + information for this phase. + </entry> + </row> + </tbody> + </tgroup> + </table> + + </sect2> + <sect2 id="vacuum-progress-reporting"> <title>VACUUM Progress Reporting</title> diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index 6e96d24ca22..54273f754f6 100644 --- a/src/backend/access/brin/brin.c +++ b/src/backend/access/brin/brin.c @@ -112,6 +112,7 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = brincostestimate; amroutine->amoptions = brinoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = brinvalidate; amroutine->ambeginscan = brinbeginscan; amroutine->amrescan = brinrescan; @@ -719,7 +720,7 @@ brinbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Now scan the relation. No syncscan allowed here because we want the * heap blocks in physical order. */ - reltuples = table_index_build_scan(heap, index, indexInfo, false, + reltuples = table_index_build_scan(heap, index, indexInfo, false, false, brinbuildCallback, (void *) state, NULL); /* process the final batch */ @@ -1236,7 +1237,7 @@ summarize_range(IndexInfo *indexInfo, BrinBuildState *state, Relation heapRel, * cases. */ state->bs_currRangeStart = heapBlk; - table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, + table_index_build_range_scan(heapRel, state->bs_irel, indexInfo, false, true, false, heapBlk, scanNumBlks, brinbuildCallback, (void *) state, NULL); diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c index b02f69b0dcb..b4b0213f76f 100644 --- a/src/backend/access/gin/gininsert.c +++ b/src/backend/access/gin/gininsert.c @@ -395,7 +395,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo) * Do the heap scan. We disallow sync scan here because dataPlaceToPage * prefers to receive tuples in TID order. */ - reltuples = table_index_build_scan(heap, index, indexInfo, false, + reltuples = table_index_build_scan(heap, index, indexInfo, false, false, ginBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c index afc20232ace..d2360eeafb0 100644 --- a/src/backend/access/gin/ginutil.c +++ b/src/backend/access/gin/ginutil.c @@ -64,6 +64,7 @@ ginhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gincostestimate; amroutine->amoptions = ginoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = ginvalidate; amroutine->ambeginscan = ginbeginscan; amroutine->amrescan = ginrescan; diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 2fddb23496d..f44c922b5d6 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -86,6 +86,7 @@ gisthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = gistcostestimate; amroutine->amoptions = gistoptions; amroutine->amproperty = gistproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = gistvalidate; amroutine->ambeginscan = gistbeginscan; amroutine->amrescan = gistrescan; diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index 3652fde5bb1..771bd2962a7 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -205,7 +205,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) /* * Do the heap scan. */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, gistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c index 5cc12a17130..6b38dd9c0e9 100644 --- a/src/backend/access/hash/hash.c +++ b/src/backend/access/hash/hash.c @@ -83,6 +83,7 @@ hashhandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = hashcostestimate; amroutine->amoptions = hashoptions; amroutine->amproperty = NULL; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = hashvalidate; amroutine->ambeginscan = hashbeginscan; amroutine->amrescan = hashrescan; @@ -160,7 +161,7 @@ hashbuild(Relation heap, Relation index, IndexInfo *indexInfo) buildstate.heapRel = heap; /* do the heap scan */ - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, hashbuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 1e4394a665b..645b532b61c 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -27,7 +27,9 @@ #include "access/xact.h" #include "catalog/catalog.h" #include "catalog/index.h" +#include "commands/progress.h" #include "executor/executor.h" +#include "pgstat.h" #include "storage/bufmgr.h" #include "storage/bufpage.h" #include "storage/lmgr.h" @@ -38,6 +40,9 @@ static const TableAmRoutine heapam_methods; +static BlockNumber heapscan_get_blocks_done(HeapScanDesc hscan); + + /* ------------------------------------------------------------------------ * Slot related callbacks for heap AM * ------------------------------------------------------------------------ @@ -529,6 +534,7 @@ heapam_index_build_range_scan(Relation heapRelation, IndexInfo *indexInfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -549,6 +555,7 @@ heapam_index_build_range_scan(Relation heapRelation, Snapshot snapshot; bool need_unregister_snapshot = false; TransactionId OldestXmin; + BlockNumber previous_blkno = InvalidBlockNumber; BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; @@ -619,6 +626,14 @@ heapam_index_build_range_scan(Relation heapRelation, NULL, /* scan key */ true, /* buffer access strategy OK */ allow_sync); /* syncscan OK? */ + + if (progress) + { + hscan = (HeapScanDesc) scan; + Assert(hscan->rs_numblocks == InvalidBlockNumber); + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + hscan->rs_nblocks); + } } else { @@ -668,6 +683,19 @@ heapam_index_build_range_scan(Relation heapRelation, CHECK_FOR_INTERRUPTS(); + /* Report scan progress, if asked to. */ + if (progress) + { + BlockNumber blocks_done = heapscan_get_blocks_done(hscan); + + if (blocks_done != previous_blkno) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blocks_done); + previous_blkno = blocks_done; + } + } + /* * When dealing with a HOT-chain of updated tuples, we want to index * the values of the live tuple (if any), but index it under the TID @@ -1042,6 +1070,7 @@ heapam_index_validate_scan(Relation heapRelation, BlockNumber root_blkno = InvalidBlockNumber; OffsetNumber root_offsets[MaxHeapTuplesPerPage]; bool in_index[MaxHeapTuplesPerPage]; + BlockNumber previous_blkno = InvalidBlockNumber; /* state variables for the merge */ ItemPointer indexcursor = NULL; @@ -1082,6 +1111,9 @@ heapam_index_validate_scan(Relation heapRelation, false); /* syncscan not OK */ hscan = (HeapScanDesc) scan; + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + hscan->rs_nblocks); + /* * Scan all tuples matching the snapshot. */ @@ -1095,6 +1127,14 @@ heapam_index_validate_scan(Relation heapRelation, state->htups += 1; + if ((previous_blkno == InvalidBlockNumber) || + (hscan->rs_cblock != previous_blkno)) + { + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + hscan->rs_cblock); + previous_blkno = hscan->rs_cblock; + } + /* * As commented in table_index_build_scan, we should index heap-only * tuples under the TIDs of their root tuples; so when we advance onto @@ -1255,6 +1295,41 @@ heapam_index_validate_scan(Relation heapRelation, indexInfo->ii_PredicateState = NULL; } +/* + * Return the number of blocks that have been read by this scan since + * starting. This is not 100% accurate: in a parallel scan, the workers + * can be concurrently reading blocks further ahead than what we report. + */ +static BlockNumber +heapscan_get_blocks_done(HeapScanDesc hscan) +{ + BlockNumber startblock; + BlockNumber blocks_done; + + if (hscan->rs_base.rs_parallel != NULL) + { + ParallelBlockTableScanDesc bpscan; + + bpscan = (ParallelBlockTableScanDesc) hscan->rs_base.rs_parallel; + startblock = bpscan->phs_startblock; + } + else + startblock = hscan->rs_startblock; + + /* + * Might have wrapped around the end of the relation, if startblock was + * not zero. + */ + if (hscan->rs_cblock > startblock) + blocks_done = hscan->rs_cblock - startblock; + else + blocks_done = hscan->rs_nblocks - startblock + + hscan->rs_cblock; + + return blocks_done; +} + + /* ------------------------------------------------------------------------ * Definition of the heap table access method. diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index ac6f1eb3423..7370379c6a1 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -22,6 +22,7 @@ #include "access/nbtxlog.h" #include "access/relscan.h" #include "access/xlog.h" +#include "commands/progress.h" #include "commands/vacuum.h" #include "miscadmin.h" #include "nodes/execnodes.h" @@ -133,6 +134,7 @@ bthandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = btcostestimate; amroutine->amoptions = btoptions; amroutine->amproperty = btproperty; + amroutine->ambuildphasename = btbuildphasename; amroutine->amvalidate = btvalidate; amroutine->ambeginscan = btbeginscan; amroutine->amrescan = btrescan; @@ -1021,6 +1023,10 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (needLock) UnlockRelationForExtension(rel, ExclusiveLock); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + num_pages); + /* Quit if we've scanned the whole relation */ if (blkno >= num_pages) break; @@ -1028,6 +1034,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, for (; blkno < num_pages; blkno++) { btvacuumpage(&vstate, blkno, blkno); + if (info->report_progress) + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE, + blkno); } } diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index a8a7b792672..b889ac6de79 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -66,6 +66,7 @@ #include "access/xlog.h" #include "access/xloginsert.h" #include "catalog/index.h" +#include "commands/progress.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/smgr.h" @@ -298,7 +299,8 @@ static double _bt_parallel_heapscan(BTBuildState *buildstate, static void _bt_leader_participate_as_worker(BTBuildState *buildstate); static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem); + Sharedsort *sharedsort2, int sortmem, + bool progress); /* @@ -394,6 +396,10 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Save as primary spool */ buildstate->spool = btspool; + /* Report heap scan phase started */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN); + /* Attempt to launch parallel worker scan when required */ if (indexInfo->ii_ParallelWorkers > 0) _bt_begin_parallel(buildstate, indexInfo->ii_Concurrent, @@ -480,13 +486,31 @@ _bt_spools_heapscan(Relation heap, Relation index, BTBuildState *buildstate, /* Fill spool using either serial or parallel heap scan */ if (!buildstate->btleader) - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, true, _bt_build_callback, (void *) buildstate, NULL); else reltuples = _bt_parallel_heapscan(buildstate, &indexInfo->ii_BrokenHotChain); + /* + * Set the progress target for the next phase. Reset the block number + * values set by IndexBuildHeapScan + */ + { + const int index[] = { + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE + }; + const int64 val[] = { + buildstate->indtuples, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } + /* okay, all heap tuples are spooled */ if (buildstate->spool2 && !buildstate->havedead) { @@ -535,9 +559,15 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) } #endif /* BTREE_BUILD_STATS */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_1); tuplesort_performsort(btspool->sortstate); if (btspool2) + { + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_PERFORMSORT_2); tuplesort_performsort(btspool2->sortstate); + } wstate.heap = btspool->heap; wstate.index = btspool->index; @@ -554,6 +584,8 @@ _bt_leafbuild(BTSpool *btspool, BTSpool *btspool2) wstate.btws_pages_written = 0; wstate.btws_zeropage = NULL; /* until needed */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_BTREE_PHASE_LEAF_LOAD); _bt_load(&wstate, btspool, btspool2); } @@ -1098,6 +1130,7 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) int i, keysz = IndexRelationGetNumberOfKeyAttributes(wstate->index); SortSupport sortKeys; + long tuples_done = 0L; if (merge) { @@ -1202,6 +1235,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) _bt_buildadd(wstate, state, itup2); itup2 = tuplesort_getindextuple(btspool2->sortstate, true); } + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } pfree(sortKeys); } @@ -1216,6 +1253,10 @@ _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) state = _bt_pagestate(wstate, 0); _bt_buildadd(wstate, state, itup); + + /* Report progress */ + pgstat_progress_update_param(PROGRESS_CREATEIDX_TUPLES_DONE, + ++tuples_done); } } @@ -1352,6 +1393,11 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request) ParallelTableScanFromBTShared(btshared), snapshot); + /* Report total number of blocks to scan */ + pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_TOTAL, + ((ParallelBlockTableScanDescData *) + ParallelTableScanFromBTShared(btshared))->phs_nblocks); + /* * Store shared tuplesort-private state, for which we reserved space. * Then, initialize opaque state using tuplesort routine. @@ -1528,7 +1574,7 @@ _bt_leader_participate_as_worker(BTBuildState *buildstate) /* Perform work common to all participants */ _bt_parallel_scan_and_sort(leaderworker, leaderworker2, btleader->btshared, btleader->sharedsort, btleader->sharedsort2, - sortmem); + sortmem, true); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1619,7 +1665,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) /* Perform sorting of spool, and possibly a spool2 */ sortmem = maintenance_work_mem / btshared->scantuplesortstates; _bt_parallel_scan_and_sort(btspool, btspool2, btshared, sharedsort, - sharedsort2, sortmem); + sharedsort2, sortmem, false); #ifdef BTREE_BUILD_STATS if (log_btree_build_stats) @@ -1648,7 +1694,7 @@ _bt_parallel_build_main(dsm_segment *seg, shm_toc *toc) static void _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, BTShared *btshared, Sharedsort *sharedsort, - Sharedsort *sharedsort2, int sortmem) + Sharedsort *sharedsort2, int sortmem, bool progress) { SortCoordinate coordinate; BTBuildState buildstate; @@ -1705,9 +1751,10 @@ _bt_parallel_scan_and_sort(BTSpool *btspool, BTSpool *btspool2, /* Join parallel scan */ indexInfo = BuildIndexInfo(btspool->index); indexInfo->ii_Concurrent = btshared->isconcurrent; - scan = table_beginscan_parallel(btspool->heap, ParallelTableScanFromBTShared(btshared)); + scan = table_beginscan_parallel(btspool->heap, + ParallelTableScanFromBTShared(btshared)); reltuples = table_index_build_scan(btspool->heap, btspool->index, indexInfo, - true, _bt_build_callback, + true, progress, _bt_build_callback, (void *) &buildstate, scan); /* diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c index 92b8b5f134d..9e8a7bd3d9a 100644 --- a/src/backend/access/nbtree/nbtutils.c +++ b/src/backend/access/nbtree/nbtutils.c @@ -20,6 +20,7 @@ #include "access/nbtree.h" #include "access/reloptions.h" #include "access/relscan.h" +#include "commands/progress.h" #include "miscadmin.h" #include "utils/array.h" #include "utils/datum.h" @@ -2048,6 +2049,29 @@ btproperty(Oid index_oid, int attno, } } +/* + * btbuildphasename() -- Return name of index build phase. + */ +char * +btbuildphasename(int64 phasenum) +{ + switch (phasenum) + { + case PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE: + return "initializing"; + case PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN: + return "table scan"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_1: + return "sorting tuples, spool 1"; + case PROGRESS_BTREE_PHASE_PERFORMSORT_2: + return "sorting tuples, spool 2"; + case PROGRESS_BTREE_PHASE_LEAF_LOAD: + return "btree tuple loading"; + default: + return NULL; + } +} + /* * _bt_truncate() -- create tuple without unneeded suffix attributes. * diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c index 390ad9ac51f..282d6998cf0 100644 --- a/src/backend/access/spgist/spginsert.c +++ b/src/backend/access/spgist/spginsert.c @@ -143,7 +143,7 @@ spgbuild(Relation heap, Relation index, IndexInfo *indexInfo) "SP-GiST build temporary context", ALLOCSET_DEFAULT_SIZES); - reltuples = table_index_build_scan(heap, index, indexInfo, true, + reltuples = table_index_build_scan(heap, index, indexInfo, true, false, spgistBuildCallback, (void *) &buildstate, NULL); diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c index 8e63c1fad25..45472db147b 100644 --- a/src/backend/access/spgist/spgutils.c +++ b/src/backend/access/spgist/spgutils.c @@ -67,6 +67,7 @@ spghandler(PG_FUNCTION_ARGS) amroutine->amcostestimate = spgcostestimate; amroutine->amoptions = spgoptions; amroutine->amproperty = spgproperty; + amroutine->ambuildphasename = NULL; amroutine->amvalidate = spgvalidate; amroutine->ambeginscan = spgbeginscan; amroutine->amrescan = spgrescan; diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 104a8cceb78..69ef1eb7bce 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -50,9 +50,9 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" -#include "commands/tablecmds.h" #include "commands/event_trigger.h" #include "commands/progress.h" +#include "commands/tablecmds.h" #include "commands/trigger.h" #include "executor/executor.h" #include "miscadmin.h" @@ -1579,7 +1579,7 @@ index_drop(Oid indexId, bool concurrent) * to acquire an exclusive lock on our table. The lock code will * detect deadlock and error out properly. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * No more predicate locks will be acquired on this index, and we're @@ -1623,7 +1623,7 @@ index_drop(Oid indexId, bool concurrent) * Wait till every transaction that saw the old index state has * finished. */ - WaitForLockers(heaplocktag, AccessExclusiveLock); + WaitForLockers(heaplocktag, AccessExclusiveLock, true); /* * Re-open relations to allow us to complete our actions. @@ -2272,6 +2272,25 @@ index_build(Relation heapRelation, save_sec_context | SECURITY_RESTRICTED_OPERATION); save_nestlevel = NewGUCNestLevel(); + /* Set up initial progress report status */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_SUBPHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_BUILD, + PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE, + 0, 0, 0, 0 + }; + + pgstat_progress_update_multi_param(6, index, val); + } + /* * Call the access method's build procedure */ @@ -2560,6 +2579,21 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) int save_sec_context; int save_nestlevel; + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_TUPLES_DONE, + PROGRESS_CREATEIDX_TUPLES_TOTAL, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN, + 0, 0, 0, 0 + }; + pgstat_progress_update_multi_param(5, index, val); + } + /* Open and lock the parent heap relation */ heapRelation = table_open(heapId, ShareUpdateExclusiveLock); /* And the target index relation */ @@ -2590,6 +2624,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) */ ivinfo.index = indexRelation; ivinfo.analyze_only = false; + ivinfo.report_progress = true; /* XXX only for btree? */ ivinfo.estimated_count = true; ivinfo.message_level = DEBUG2; ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples; @@ -2607,15 +2642,39 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot) NULL, false); state.htups = state.itups = state.tups_inserted = 0; + /* ambulkdelete updates progress metrics */ (void) index_bulk_delete(&ivinfo, NULL, validate_index_callback, (void *) &state); /* Execute the sort */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + PROGRESS_SCAN_BLOCKS_DONE, + PROGRESS_SCAN_BLOCKS_TOTAL + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN, + 0, 0 + }; + + pgstat_progress_update_multi_param(3, index, val); + } tuplesort_performsort(state.tuplesort); /* - * Now scan the heap and "merge" it with the index + * Now scan the heap and "merge" it with the index. */ + { + const int index[] = { + PROGRESS_CREATEIDX_PHASE, + }; + const int64 val[] = { + PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE + }; + + pgstat_progress_update_multi_param(1, index, val); + } table_index_validate_scan(heapRelation, indexRelation, indexInfo, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index b89df70653e..b7c040d5284 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -934,6 +934,33 @@ CREATE VIEW pg_stat_progress_cluster AS FROM pg_stat_get_progress_info('CLUSTER') AS S LEFT JOIN pg_database D ON S.datid = D.oid; +CREATE VIEW pg_stat_progress_create_index AS + SELECT + S.pid AS pid, S.datid AS datid, D.datname AS datname, + S.relid AS relid, + CASE S.param2 WHEN 0 THEN 'initializing' + WHEN 1 THEN 'waiting for old snapshots' + WHEN 2 THEN 'building index' || + COALESCE((': ' || pg_indexam_progress_phasename(S.param1::oid, S.param3)), + '') + WHEN 3 THEN 'waiting for writer snapshots' + WHEN 4 THEN 'index validation: scan index' + WHEN 5 THEN 'index validation: sort index scan results' + WHEN 6 THEN 'index validation: scan heap' + WHEN 7 THEN 'waiting for reader snapshots' + END as phase, + S.param4 AS lockers_total, + S.param5 AS lockers_done, + S.param6 AS current_locker_pid, + S.param7 AS blocks_total, + S.param8 AS blocks_done, + S.param9 AS tuples_total, + S.param10 AS tuples_done, + S.param11 AS partitions_total, + S.param12 AS partitions_done + FROM pg_stat_get_progress_info('CREATE INDEX') AS S + LEFT JOIN pg_database D ON S.datid = D.oid; + CREATE VIEW pg_user_mappings AS SELECT U.oid AS umid, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index d6eb48cb4e6..10c6f716455 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -36,6 +36,7 @@ #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" +#include "commands/progress.h" #include "commands/tablecmds.h" #include "commands/tablespace.h" #include "mb/pg_wchar.h" @@ -47,10 +48,12 @@ #include "parser/parse_func.h" #include "parser/parse_oper.h" #include "partitioning/partdesc.h" +#include "pgstat.h" #include "rewrite/rewriteManip.h" #include "storage/lmgr.h" #include "storage/proc.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/acl.h" #include "utils/builtins.h" #include "utils/fmgroids.h" @@ -369,6 +372,15 @@ DefineIndex(Oid relationId, Snapshot snapshot; int i; + + /* + * Start progress report. If we're building a partition, this was already + * done. + */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX, + relationId); + /* * count key attributes in index */ @@ -585,6 +597,9 @@ DefineIndex(Oid relationId, accessMethodId = accessMethodForm->oid; amRoutine = GetIndexAmRoutine(accessMethodForm->amhandler); + pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID, + accessMethodId); + if (stmt->unique && !amRoutine->amcanunique) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -865,6 +880,11 @@ DefineIndex(Oid relationId, if (!OidIsValid(indexRelationId)) { table_close(rel, NoLock); + + /* If this is the top-level index, we're done */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -890,6 +910,9 @@ DefineIndex(Oid relationId, TupleDesc parentDesc; Oid *opfamOids; + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL, + nparts); + memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts); parentDesc = CreateTupleDescCopy(RelationGetDescr(rel)); @@ -1039,6 +1062,8 @@ DefineIndex(Oid relationId, skip_build, quiet); } + pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, + i + 1); pfree(attmap); } @@ -1073,6 +1098,8 @@ DefineIndex(Oid relationId, * Indexes on partitioned tables are not themselves built, so we're * done here. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); return address; } @@ -1080,6 +1107,11 @@ DefineIndex(Oid relationId, { /* Close the heap and we're done, in the non-concurrent case */ table_close(rel, NoLock); + + /* If this is the top-level index, we're done. */ + if (!OidIsValid(parentIndexId)) + pgstat_progress_end_command(); + return address; } @@ -1131,7 +1163,9 @@ DefineIndex(Oid relationId, * exclusive lock on our table. The lock code will detect deadlock and * error out properly. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_1); + WaitForLockers(heaplocktag, ShareLock, true); /* * At this moment we are sure that there are no transactions with the @@ -1195,7 +1229,9 @@ DefineIndex(Oid relationId, * We once again wait until no transaction can have the table open with * the index marked as read-only for updates. */ - WaitForLockers(heaplocktag, ShareLock); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_2); + WaitForLockers(heaplocktag, ShareLock, true); /* * Now take the "reference snapshot" that will be used by validate_index() @@ -1281,6 +1317,9 @@ DefineIndex(Oid relationId, old_snapshots = GetCurrentVirtualXIDs(limitXmin, true, false, PROC_IS_AUTOVACUUM | PROC_IN_VACUUM, &n_old_snapshots); + pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE, + PROGRESS_CREATEIDX_PHASE_WAIT_3); + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, n_old_snapshots); for (i = 0; i < n_old_snapshots; i++) { @@ -1316,7 +1355,14 @@ DefineIndex(Oid relationId, } if (VirtualTransactionIdIsValid(old_snapshots[i])) + { + PGPROC *holder = BackendIdGetProc(old_snapshots[i].backendId); + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); VirtualXactLock(old_snapshots[i], true); + } + + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1); } /* @@ -1339,6 +1385,8 @@ DefineIndex(Oid relationId, */ UnlockRelationIdForSession(&heaprelid, ShareUpdateExclusiveLock); + pgstat_progress_end_command(); + return address; } diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index cd56dca3aef..215f1463bb1 100644 --- a/src/backend/storage/ipc/standby.c +++ b/src/backend/storage/ipc/standby.c @@ -401,7 +401,7 @@ ResolveRecoveryConflictWithLock(LOCKTAG locktag) */ VirtualTransactionId *backends; - backends = GetLockConflicts(&locktag, AccessExclusiveLock); + backends = GetLockConflicts(&locktag, AccessExclusiveLock, NULL); ResolveRecoveryConflictWithVirtualXIDs(backends, PROCSIG_RECOVERY_CONFLICT_LOCK); } diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c index e688ba81170..0b04b093782 100644 --- a/src/backend/storage/lmgr/lmgr.c +++ b/src/backend/storage/lmgr/lmgr.c @@ -19,9 +19,12 @@ #include "access/transam.h" #include "access/xact.h" #include "catalog/catalog.h" +#include "commands/progress.h" #include "miscadmin.h" +#include "pgstat.h" #include "storage/lmgr.h" #include "storage/procarray.h" +#include "storage/sinvaladt.h" #include "utils/inval.h" @@ -857,10 +860,12 @@ XactLockTableWaitErrorCb(void *arg) * after we obtained our initial list of lockers, we will not wait for them. */ void -WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) +WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress) { List *holders = NIL; ListCell *lc; + int total = 0; + int done = 0; /* Done if no locks to wait for */ if (list_length(locktags) == 0) @@ -870,10 +875,17 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) foreach(lc, locktags) { LOCKTAG *locktag = lfirst(lc); + int count; - holders = lappend(holders, GetLockConflicts(locktag, lockmode)); + holders = lappend(holders, + GetLockConflicts(locktag, lockmode, + progress ? &count : NULL)); + total += count; } + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total); + /* * Note: GetLockConflicts() never reports our own xid, hence we need not * check for that. Also, prepared xacts are not reported, which is fine @@ -887,10 +899,36 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) while (VirtualTransactionIdIsValid(*lockholders)) { + /* + * If requested, publish who we're going to wait for. This is not + * 100% accurate if they're already gone, but we don't care. + */ + if (progress) + { + PGPROC *holder = BackendIdGetProc(lockholders->backendId); + + pgstat_progress_update_param(PROGRESS_WAITFOR_CURRENT_PID, + holder->pid); + } VirtualXactLock(*lockholders, true); lockholders++; + + if (progress) + pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done); } } + if (progress) + { + const int index[] = { + PROGRESS_WAITFOR_TOTAL, + PROGRESS_WAITFOR_DONE, + PROGRESS_WAITFOR_CURRENT_PID + }; + const int64 values[] = { + 0, 0, 0 + }; + pgstat_progress_update_multi_param(3, index, values); + } list_free_deep(holders); } @@ -901,12 +939,12 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode) * Same as WaitForLockersMultiple, for a single lock tag. */ void -WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode) +WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress) { List *l; l = list_make1(&heaplocktag); - WaitForLockersMultiple(l, lockmode); + WaitForLockersMultiple(l, lockmode, progress); list_free(l); } diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c index 78fdbd6ff88..c8958766f1e 100644 --- a/src/backend/storage/lmgr/lock.c +++ b/src/backend/storage/lmgr/lock.c @@ -2807,6 +2807,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * xacts merely awaiting such a lock are NOT reported. * * The result array is palloc'd and is terminated with an invalid VXID. + * *countp, if not null, is updated to the number of items set. * * Of course, the result could be out of date by the time it's returned, * so use of this function has to be thought about carefully. @@ -2817,7 +2818,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock) * uses of the result. */ VirtualTransactionId * -GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) +GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp) { static VirtualTransactionId *vxids; LOCKMETHODID lockmethodid = locktag->locktag_lockmethodid; @@ -2964,6 +2965,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) LWLockRelease(partitionLock); vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } @@ -3019,6 +3022,8 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode) vxids[count].backendId = InvalidBackendId; vxids[count].localTransactionId = InvalidLocalTransactionId; + if (countp) + *countp = count; return vxids; } diff --git a/src/backend/utils/adt/amutils.c b/src/backend/utils/adt/amutils.c index 060ffe501ec..e81d6cc0562 100644 --- a/src/backend/utils/adt/amutils.c +++ b/src/backend/utils/adt/amutils.c @@ -445,3 +445,26 @@ pg_index_column_has_property(PG_FUNCTION_ARGS) return indexam_property(fcinfo, propname, InvalidOid, relid, attno); } + +/* + * Return the name of the given phase, as used for progress reporting by the + * given AM. + */ +Datum +pg_indexam_progress_phasename(PG_FUNCTION_ARGS) +{ + Oid amoid = PG_GETARG_OID(0); + int32 phasenum = PG_GETARG_INT32(1); + IndexAmRoutine *routine; + char *name; + + routine = GetIndexAmRoutineByAmId(amoid, true); + if (routine == NULL || !routine->ambuildphasename) + PG_RETURN_NULL(); + + name = routine->ambuildphasename(phasenum); + if (!name) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(CStringGetTextDatum(name)); +} diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 90a817a25c5..7c2afe64272 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -470,6 +470,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS) cmdtype = PROGRESS_COMMAND_VACUUM; else if (pg_strcasecmp(cmd, "CLUSTER") == 0) cmdtype = PROGRESS_COMMAND_CLUSTER; + else if (pg_strcasecmp(cmd, "CREATE INDEX") == 0) + cmdtype = PROGRESS_COMMAND_CREATE_INDEX; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index 653ddc976ba..09a7404267c 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -108,6 +108,9 @@ typedef bool (*amproperty_function) (Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +/* name of phase as used in progress reporting */ +typedef char *(*ambuildphasename_function) (int64 phasenum); + /* validate definition of an opclass for this AM */ typedef bool (*amvalidate_function) (Oid opclassoid); @@ -213,6 +216,7 @@ typedef struct IndexAmRoutine amcostestimate_function amcostestimate; amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ + ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; ambeginscan_function ambeginscan; amrescan_function amrescan; diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 70c7351a08c..9717183ef23 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -45,6 +45,7 @@ typedef struct IndexVacuumInfo { Relation index; /* the index being vacuumed */ bool analyze_only; /* ANALYZE (without any actual vacuum) */ + bool report_progress; /* emit progress.h status reports */ bool estimated_count; /* num_heap_tuples is an estimate */ int message_level; /* ereport level for progress messages */ double num_heap_tuples; /* tuples remaining in heap */ diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h index 473c6f29185..bc0f3027629 100644 --- a/src/include/access/nbtree.h +++ b/src/include/access/nbtree.h @@ -672,6 +672,16 @@ typedef BTScanOpaqueData *BTScanOpaque; #define SK_BT_DESC (INDOPTION_DESC << SK_BT_INDOPTION_SHIFT) #define SK_BT_NULLS_FIRST (INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT) +/* + * Constant definition for progress reporting. Phase numbers must match + * btbuildphasename. + */ +/* PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE is 1 (see progress.h) */ +#define PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN 2 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_1 3 +#define PROGRESS_BTREE_PHASE_PERFORMSORT_2 4 +#define PROGRESS_BTREE_PHASE_LEAF_LOAD 5 + /* * external entry points for btree, in nbtree.c */ @@ -785,6 +795,7 @@ extern bytea *btoptions(Datum reloptions, bool validate); extern bool btproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +extern char *btbuildphasename(int64 phasenum); extern IndexTuple _bt_truncate(Relation rel, IndexTuple lastleft, IndexTuple firstright, BTScanInsert itup_key); extern int _bt_keep_natts_fast(Relation rel, IndexTuple lastleft, diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 2546d3005fb..9579006d447 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -384,6 +384,7 @@ typedef struct TableAmRoutine struct IndexInfo *index_nfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber end_blockno, IndexBuildCallback callback, @@ -978,6 +979,11 @@ table_lock_tuple(Relation rel, ItemPointer tid, Snapshot snapshot, * so here because the AM might reject some of the tuples for its own reasons, * such as being unable to store NULLs. * + * If 'progress', we update the PROGRESS_SCAN_BLOCKS_DONE counter as we go + * along. Also, if that flag is true and a scan descriptor is not passed (ie. + * when not doing a parallel scan), the PROGRESS_SCAN_BLOCKS_TOTAL counter is + * updated at the beginning. For parallel scans, caller is expected to have + * set the total number of blocks prior to calling this function. * * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect * any potentially broken HOT chains. Currently, we set this if there are any @@ -991,6 +997,7 @@ table_index_build_scan(Relation heap_rel, Relation index_rel, struct IndexInfo *index_nfo, bool allow_sync, + bool progress, IndexBuildCallback callback, void *callback_state, TableScanDesc scan) @@ -1000,6 +1007,7 @@ table_index_build_scan(Relation heap_rel, index_nfo, allow_sync, false, + progress, 0, InvalidBlockNumber, callback, @@ -1023,6 +1031,7 @@ table_index_build_range_scan(Relation heap_rel, struct IndexInfo *index_nfo, bool allow_sync, bool anyvisible, + bool progress, BlockNumber start_blockno, BlockNumber numblocks, IndexBuildCallback callback, @@ -1034,6 +1043,7 @@ table_index_build_range_scan(Relation heap_rel, index_nfo, allow_sync, anyvisible, + progress, start_blockno, numblocks, callback, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index eac909109c5..fcd6d743bea 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -924,6 +924,10 @@ proname => 'pg_index_column_has_property', provolatile => 's', prorettype => 'bool', proargtypes => 'regclass int4 text', prosrc => 'pg_index_column_has_property' }, +{ oid => '676', descr => 'return name of given index build phase', + proname => 'pg_indexam_progress_phasename', provolatile => 'i', + prorettype => 'text', proargtypes => 'oid int8', + prosrc => 'pg_indexam_progress_phasename' }, { oid => '339', proname => 'poly_same', prorettype => 'bool', @@ -5122,9 +5126,9 @@ proname => 'pg_stat_get_progress_info', prorows => '100', proretset => 't', provolatile => 's', proparallel => 'r', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', - proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}', + proallargtypes => '{text,int4,oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10,param11,param12}', prosrc => 'pg_stat_get_progress_info' }, { oid => '3099', descr => 'statistics: information about currently active replication', diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h index 04542d9e923..45847f6dc35 100644 --- a/src/include/commands/progress.h +++ b/src/include/commands/progress.h @@ -57,4 +57,39 @@ #define PROGRESS_CLUSTER_COMMAND_CLUSTER 1 #define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL 2 +/* Progress parameters for CREATE INDEX */ +#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID 0 +#define PROGRESS_CREATEIDX_PHASE 1 /* AM-agnostic phase # */ +#define PROGRESS_CREATEIDX_SUBPHASE 2 /* phase # filled by AM */ +/* 3, 4 and 5 reserved for "waitfor" metrics */ +/* 6 and 7 reserved for "block number" metrics */ +#define PROGRESS_CREATEIDX_TUPLES_TOTAL 8 +#define PROGRESS_CREATEIDX_TUPLES_DONE 9 +#define PROGRESS_CREATEIDX_PARTITIONS_TOTAL 10 +#define PROGRESS_CREATEIDX_PARTITIONS_DONE 11 + +/* Phases of CREATE INDEX */ +#define PROGRESS_CREATEIDX_PHASE_WAIT_1 1 +#define PROGRESS_CREATEIDX_PHASE_BUILD 2 +#define PROGRESS_CREATEIDX_PHASE_WAIT_2 3 +#define PROGRESS_CREATEIDX_PHASE_VALIDATE_IDXSCAN 4 +#define PROGRESS_CREATEIDX_PHASE_SORT_IDXSCAN 5 +#define PROGRESS_CREATEIDX_PHASE_HEAPSCAN_VALIDATE 6 +#define PROGRESS_CREATEIDX_PHASE_WAIT_3 7 + +/* + * Subphases of CREATE INDEX, for index_build. + */ +#define PROGRESS_CREATEIDX_SUBPHASE_INITIALIZE 1 +/* Additional phases are defined by each AM */ + +/* Lock holder wait counts */ +#define PROGRESS_WAITFOR_TOTAL 3 +#define PROGRESS_WAITFOR_DONE 4 +#define PROGRESS_WAITFOR_CURRENT_PID 5 + +/* Block numbers in a generic relation scan */ +#define PROGRESS_SCAN_BLOCKS_TOTAL 6 +#define PROGRESS_SCAN_BLOCKS_DONE 7 + #endif diff --git a/src/include/pgstat.h b/src/include/pgstat.h index c080fa6388f..6af9ab7e1d9 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -951,10 +951,11 @@ typedef enum ProgressCommandType { PROGRESS_COMMAND_INVALID, PROGRESS_COMMAND_VACUUM, - PROGRESS_COMMAND_CLUSTER + PROGRESS_COMMAND_CLUSTER, + PROGRESS_COMMAND_CREATE_INDEX } ProgressCommandType; -#define PGSTAT_NUM_PROGRESS_PARAM 10 +#define PGSTAT_NUM_PROGRESS_PARAM 12 /* ---------- * Shared-memory data structures diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h index 3d705faba5c..4f2872de35f 100644 --- a/src/include/storage/lmgr.h +++ b/src/include/storage/lmgr.h @@ -78,8 +78,8 @@ extern void XactLockTableWait(TransactionId xid, Relation rel, extern bool ConditionalXactLockTableWait(TransactionId xid); /* Lock VXIDs, specified by conflicting locktags */ -extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode); -extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode); +extern void WaitForLockers(LOCKTAG heaplocktag, LOCKMODE lockmode, bool progress); +extern void WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress); /* Lock an XID for tuple insertion (used to wait for an insertion to finish) */ extern uint32 SpeculativeInsertionLockAcquire(TransactionId xid); diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h index badf7fd682b..048947c50d4 100644 --- a/src/include/storage/lock.h +++ b/src/include/storage/lock.h @@ -544,7 +544,7 @@ extern bool LockHeldByMe(const LOCKTAG *locktag, LOCKMODE lockmode); extern bool LockHasWaiters(const LOCKTAG *locktag, LOCKMODE lockmode, bool sessionLock); extern VirtualTransactionId *GetLockConflicts(const LOCKTAG *locktag, - LOCKMODE lockmode); + LOCKMODE lockmode, int *countp); extern void AtPrepare_Locks(void); extern void PostPrepare_Locks(TransactionId xid); extern int LockCheckConflicts(LockMethod lockMethodTable, diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index d5f309fbfbe..a9a2910fb55 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1856,7 +1856,33 @@ pg_stat_progress_cluster| SELECT s.pid, s.param6 AS heap_blks_total, s.param7 AS heap_blks_scanned, s.param8 AS index_rebuild_count - FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) + LEFT JOIN pg_database d ON ((s.datid = d.oid))); +pg_stat_progress_create_index| SELECT s.pid, + s.datid, + d.datname, + s.relid, + CASE s.param2 + WHEN 0 THEN 'initializing'::text + WHEN 1 THEN 'waiting for old snapshots'::text + WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param1)::oid, s.param3)), ''::text)) + WHEN 3 THEN 'waiting for writer snapshots'::text + WHEN 4 THEN 'index validation: scan index'::text + WHEN 5 THEN 'index validation: sort index scan results'::text + WHEN 6 THEN 'index validation: scan heap'::text + WHEN 7 THEN 'waiting for reader snapshots'::text + ELSE NULL::text + END AS phase, + s.param4 AS lockers_total, + s.param5 AS lockers_done, + s.param6 AS current_locker_pid, + s.param7 AS blocks_total, + s.param8 AS blocks_done, + s.param9 AS tuples_total, + s.param10 AS tuples_done, + s.param11 AS partitions_total, + s.param12 AS partitions_done + FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_progress_vacuum| SELECT s.pid, s.datid, @@ -1878,7 +1904,7 @@ pg_stat_progress_vacuum| SELECT s.pid, s.param5 AS index_vacuum_count, s.param6 AS max_dead_tuples, s.param7 AS num_dead_tuples - FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10) + FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12) LEFT JOIN pg_database d ON ((s.datid = d.oid))); pg_stat_replication| SELECT s.pid, s.usesysid, -- 2.17.1 --SUOF0GtieIMvvwua Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-report-partial-progress-of-other-index-AMs.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v3 1/1] Change output of ECHO_HIDDEN comments to be SQL comments @ 2023-05-17 20:59 Kirk Wolak <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Kirk Wolak @ 2023-05-17 20:59 UTC (permalink / raw) Simply Add the "/" Required before/after the ********* QUERY ********** (SHOW_HIDDEN on) so that they are comments and are ignored if you copy and paste a series of them. Author: Kirk Wolak <[email protected]> Reviewed By: Pavel Stehule <[email protected]> Reviewed By: Laurenz Albe <[email protected]> Thread: https://postgr.es/m/CACLU5mTFJRJYtbvmZ26txGgmXWQo0hkGhH2o3hEquUPmSbGtBw@mail.gmail.com --- src/bin/psql/command.c | 8 ++++---- src/bin/psql/common.c | 16 ++++++++-------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c index 6733f008fd..1300869d79 100644 --- a/src/bin/psql/command.c +++ b/src/bin/psql/command.c @@ -5395,16 +5395,16 @@ echo_hidden_command(const char *query) { if (pset.echo_hidden != PSQL_ECHO_HIDDEN_OFF) { - printf(_("********* QUERY **********\n" + printf(_("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(stdout); if (pset.logfile) { fprintf(pset.logfile, - _("********* QUERY **********\n" + _("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(pset.logfile); } diff --git a/src/bin/psql/common.c b/src/bin/psql/common.c index 5973df2e39..10ad1f2538 100644 --- a/src/bin/psql/common.c +++ b/src/bin/psql/common.c @@ -589,16 +589,16 @@ PSQLexec(const char *query) if (pset.echo_hidden != PSQL_ECHO_HIDDEN_OFF) { - printf(_("********* QUERY **********\n" + printf(_("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(stdout); if (pset.logfile) { fprintf(pset.logfile, - _("********* QUERY **********\n" + _("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(pset.logfile); } @@ -1060,9 +1060,9 @@ SendQuery(const char *query) char buf[3]; fflush(stderr); - printf(_("***(Single step mode: verify command)*******************************************\n" + printf(_("/**(Single step mode: verify command)******************************************/\n" "%s\n" - "***(press return to proceed or enter x and return to cancel)********************\n"), + "/**(press return to proceed or enter x and return to cancel)*******************/\n"), query); fflush(stdout); if (fgets(buf, sizeof(buf), stdin) != NULL) @@ -1080,9 +1080,9 @@ SendQuery(const char *query) if (pset.logfile) { fprintf(pset.logfile, - _("********* QUERY **********\n" + _("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(pset.logfile); } -- 2.25.1 --Qxx1br4bt0+wmkIi-- ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v3 1/1] Change output of ECHO_HIDDEN comments to be SQL comments @ 2023-05-17 20:59 Kirk Wolak <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Kirk Wolak @ 2023-05-17 20:59 UTC (permalink / raw) Simply Add the "/" Required before/after the ********* QUERY ********** (SHOW_HIDDEN on) so that they are comments and are ignored if you copy and paste a series of them. Author: Kirk Wolak <[email protected]> Reviewed By: Pavel Stehule <[email protected]> Reviewed By: Laurenz Albe <[email protected]> Thread: https://postgr.es/m/CACLU5mTFJRJYtbvmZ26txGgmXWQo0hkGhH2o3hEquUPmSbGtBw@mail.gmail.com --- src/bin/psql/command.c | 8 ++++---- src/bin/psql/common.c | 16 ++++++++-------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c index 6733f008fd..1300869d79 100644 --- a/src/bin/psql/command.c +++ b/src/bin/psql/command.c @@ -5395,16 +5395,16 @@ echo_hidden_command(const char *query) { if (pset.echo_hidden != PSQL_ECHO_HIDDEN_OFF) { - printf(_("********* QUERY **********\n" + printf(_("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(stdout); if (pset.logfile) { fprintf(pset.logfile, - _("********* QUERY **********\n" + _("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(pset.logfile); } diff --git a/src/bin/psql/common.c b/src/bin/psql/common.c index 5973df2e39..10ad1f2538 100644 --- a/src/bin/psql/common.c +++ b/src/bin/psql/common.c @@ -589,16 +589,16 @@ PSQLexec(const char *query) if (pset.echo_hidden != PSQL_ECHO_HIDDEN_OFF) { - printf(_("********* QUERY **********\n" + printf(_("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(stdout); if (pset.logfile) { fprintf(pset.logfile, - _("********* QUERY **********\n" + _("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(pset.logfile); } @@ -1060,9 +1060,9 @@ SendQuery(const char *query) char buf[3]; fflush(stderr); - printf(_("***(Single step mode: verify command)*******************************************\n" + printf(_("/**(Single step mode: verify command)******************************************/\n" "%s\n" - "***(press return to proceed or enter x and return to cancel)********************\n"), + "/**(press return to proceed or enter x and return to cancel)*******************/\n"), query); fflush(stdout); if (fgets(buf, sizeof(buf), stdin) != NULL) @@ -1080,9 +1080,9 @@ SendQuery(const char *query) if (pset.logfile) { fprintf(pset.logfile, - _("********* QUERY **********\n" + _("/******** QUERY *********/\n" "%s\n" - "**************************\n\n"), query); + "/************************/\n\n"), query); fflush(pset.logfile); } -- 2.25.1 --Qxx1br4bt0+wmkIi-- ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib) @ 2025-01-07 18:25 Darek Ślusarczyk <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Darek Ślusarczyk @ 2025-01-07 18:25 UTC (permalink / raw) To: Nazir Bilal Yavuz <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Daniel Gustafsson <[email protected]> Hi, Thanks for reviewing the patch - I've added a few changes and it looks as follows: diff --git a/meson.build b/meson.build index e5ce437a5c7..37d4e9ca741 100644 --- a/meson.build +++ b/meson.build @@ -1343,14 +1343,41 @@ if sslopt in ['auto', 'openssl'] # via library + headers if not ssl.found() + is_windows = host_system == 'windows' + + ssl_lib_common_params = { + 'dirs': test_lib_d, + 'header_include_directories': postgres_inc, + 'has_headers': ['openssl/ssl.h', 'openssl/err.h'], + } ssl_lib = cc.find_library('ssl', - dirs: test_lib_d, - header_include_directories: postgres_inc, - has_headers: ['openssl/ssl.h', 'openssl/err.h'], - required: openssl_required) + kwargs: ssl_lib_common_params, + required: openssl_required and not is_windows + ) + # before openssl version 1.1.0, there was a different naming convention for libraries on win + # the counterpart for libssl.[lib|dll] was ssleay32.[lib|dll] + # more details in https://github.com/openssl/openssl/issues/10332#issuecomment-549027653 + # hence if 'ssl' is not found and it's windows, try 'ssleay32' + if not ssl_lib.found() and is_windows + ssl_lib = cc.find_library('ssleay32', + kwargs: ssl_lib_common_params, + required: openssl_required + ) + endif + crypto_lib = cc.find_library('crypto', dirs: test_lib_d, - required: openssl_required) + required: openssl_required and not is_windows) + # before openssl version 1.1.0, there was a different naming convention for libraries on win + # the counterpart for libcrypto.[lib|dll] was libeay32.[lib|dll] + # more details in https://github.com/openssl/openssl/issues/10332#issuecomment-549027653 + # hence if 'crypto' is not found and it's windows, try 'libeay32' + if not crypto_lib.found() and is_windows + crypto_lib = cc.find_library('libeay32', + dirs: test_lib_d, + required: openssl_required) + endif + if ssl_lib.found() and crypto_lib.found() ssl_int = [ssl_lib, crypto_lib] ssl = declare_dependency(dependencies: ssl_int, include_directories: postgres_inc) btw I also submitted a pull request on GitHub, which can be found here: https://github.com/postgres/postgres/pull/197 thnx, ds -- marines() { return Darek_Slusarczyk; } ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib) @ 2025-01-09 12:50 Daniel Gustafsson <[email protected]> parent: Darek Ślusarczyk <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Daniel Gustafsson @ 2025-01-09 12:50 UTC (permalink / raw) To: Darek Ślusarczyk <[email protected]>; +Cc: Nazir Bilal Yavuz <[email protected]>; PostgreSQL Hackers <[email protected]> > On 7 Jan 2025, at 19:25, Darek Ślusarczyk <[email protected]> wrote: > Thanks for reviewing the patch - I've added a few changes and it looks as follows: Thanks for the new version, I've tested it in CI against v17 and plan to commit it soon. -- Daniel Gustafsson ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib) @ 2025-02-07 14:14 Daniel Gustafsson <[email protected]> parent: Daniel Gustafsson <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Daniel Gustafsson @ 2025-02-07 14:14 UTC (permalink / raw) To: Darek Ślusarczyk <[email protected]>; +Cc: Nazir Bilal Yavuz <[email protected]>; PostgreSQL Hackers <[email protected]> > On 9 Jan 2025, at 13:50, Daniel Gustafsson <[email protected]> wrote: > >> On 7 Jan 2025, at 19:25, Darek Ślusarczyk <[email protected]> wrote: > >> Thanks for reviewing the patch - I've added a few changes and it looks as follows: > > Thanks for the new version, I've tested it in CI against v17 and plan to commit > it soon. Committed to v17 and backpatched to v16 which is when the Meson support was added. -- Daniel Gustafsson ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib) @ 2025-02-07 14:20 Darek Ślusarczyk <[email protected]> parent: Daniel Gustafsson <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Darek Ślusarczyk @ 2025-02-07 14:20 UTC (permalink / raw) To: Daniel Gustafsson <[email protected]>; +Cc: Nazir Bilal Yavuz <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Feb 7, 2025 at 3:14 PM Daniel Gustafsson <[email protected]> wrote: > > On 9 Jan 2025, at 13:50, Daniel Gustafsson <[email protected]> wrote: > > > >> On 7 Jan 2025, at 19:25, Darek Ślusarczyk <[email protected]> > wrote: > > > >> Thanks for reviewing the patch - I've added a few changes and it looks > as follows: > > > > Thanks for the new version, I've tested it in CI against v17 and plan to > commit > > it soon. > > Committed to v17 and backpatched to v16 which is when the Meson support was > added. > Great news, thnx! -- marines() { return Darek_Slusarczyk; } ^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2025-02-07 14:20 UTC | newest] Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2019-01-02 19:14 [PATCH] Report progress of CREATE INDEX operations Alvaro Herrera <[email protected]> 2019-01-02 19:14 [PATCH v6 2/3] Report progress of CREATE INDEX operations Alvaro Herrera <[email protected]> 2019-01-02 19:14 [PATCH v5 1/2] Report progress of CREATE INDEX operations Alvaro Herrera <[email protected]> 2019-03-25 16:29 [PATCH v8 1/2] Report progress of CREATE INDEX operations Alvaro Herrera <[email protected]> 2019-03-25 16:29 [PATCH v9 1/2] Report progress of CREATE INDEX operations Alvaro Herrera <[email protected]> 2019-03-25 16:29 [PATCH v7 1/2] Report progress of CREATE INDEX operations Alvaro Herrera <[email protected]> 2023-05-17 20:59 [PATCH v3 1/1] Change output of ECHO_HIDDEN comments to be SQL comments Kirk Wolak <[email protected]> 2023-05-17 20:59 [PATCH v3 1/1] Change output of ECHO_HIDDEN comments to be SQL comments Kirk Wolak <[email protected]> 2025-01-07 18:25 Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib) Darek Ślusarczyk <[email protected]> 2025-01-09 12:50 ` Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib) Daniel Gustafsson <[email protected]> 2025-02-07 14:14 ` Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib) Daniel Gustafsson <[email protected]> 2025-02-07 14:20 ` Re: add support for the old naming libs convention on windows (ssleay32.lib and libeay32.lib) Darek Ślusarczyk <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox