public inbox for [email protected]
help / color / mirror / Atom feedFrom: Imseih (AWS), Sami <[email protected]>
To: Masahiko Sawada <[email protected]>
Cc: Robert Haas <[email protected]>
Cc: Bossart, Nathan <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Peter Geoghegan <[email protected]>
Cc: Justin Pryzby <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: Add index scan progress to pg_stat_progress_vacuum
Date: Thu, 26 May 2022 13:41:09 +0000
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<CAD21AoD13LQ4+suYVWCyp07PHgoHVmAPZwc8neNdkTFVsSi-ww@mail.gmail.com>
<[email protected]>
<CAD21AoAwsaohDMissr_fR_YHp3fzq+eJ4n2=G2Bxji8aMqPAxA@mail.gmail.com>
<[email protected]>
<CAD21AoDAj67or-qEMZfdtz74=dgEpt1D=6ZcGJMK8TGvpR38aQ@mail.gmail.com>
<[email protected]>
<CAD21AoA9Phn9uPtyq_V+J9ctNhkgChnmm_hFT2sA2qR_KcKqYw@mail.gmail.com>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<CA+TgmoaQDGtKs8qotVBHRzr-G5OnrQNdeknAjM_GKdc0J_ideg@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAD21AoATKO4Ltizm2CXtggE081tYeJSDWngz0w4w13wTwwCDpA@mail.gmail.com>
<[email protected]>
> the function after updating the index status to
> PARALLEL_INDVAC_STATUS_COMPLETED.
> I also like this better. Will make the change.
I updated the patch. The progress function is called after
updating index status to PARALLEL_INDVAC_STATUS_COMPLETED.
I believe all comments have been addressed at this point.
Regards,
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v11-0001-Add-progress-reporting-callback-to-ParallelConte.patch (2.6K, ../[email protected]/2-v11-0001-Add-progress-reporting-callback-to-ParallelConte.patch)
download | inline diff:
From 19b1a9b4814fdf411a34a0ca86ce224079c857cc Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Thu, 26 May 2022 07:45:26 -0500
Subject: [PATCH v11 1/2] Add progress reporting callback to ParallelContext
The purpose of supporting a progress reporting
callback in ParallelContext is to allow for the
leader process to report progress while waiting
for workers to complete.
The first use-case for this is to report index
progress in pg_stat_progress_vacuum.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
src/backend/access/transam/parallel.c | 16 ++++++++++++++++
src/include/access/parallel.h | 5 +++++
2 files changed, 21 insertions(+)
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index df0cd77..bfe3275 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -774,6 +774,22 @@ WaitForParallelWorkersToFinish(ParallelContext *pcxt)
*/
CHECK_FOR_INTERRUPTS();
+ /*
+ * We call the parallel progress callback while
+ * waiting for the parallel workers to finish.
+ * This is to ensure that the leader keeps
+ * updating progress when waiting for
+ * parallel workers to finish.
+ *
+ * We must ensure that pcxt->parallel_progress_callback
+ * is set before calling as not all parallel
+ * operations will set a callback.
+ */
+ if (pcxt->parallel_progress_callback)
+ {
+ pcxt->parallel_progress_callback(pcxt->parallel_progress_callback_arg);
+ }
+
for (i = 0; i < pcxt->nworkers_launched; ++i)
{
/*
diff --git a/src/include/access/parallel.h b/src/include/access/parallel.h
index 983841d..53a3d13 100644
--- a/src/include/access/parallel.h
+++ b/src/include/access/parallel.h
@@ -20,6 +20,9 @@
#include "storage/shm_mq.h"
#include "storage/shm_toc.h"
+/* progress callback definition */
+typedef void (*ParallelProgressCallback) (void *parallel_progress_callback_state);
+
typedef void (*parallel_worker_main_type) (dsm_segment *seg, shm_toc *toc);
typedef struct ParallelWorkerInfo
@@ -46,6 +49,8 @@ typedef struct ParallelContext
ParallelWorkerInfo *worker;
int nknown_attached_workers;
bool *known_attached_workers;
+ ParallelProgressCallback parallel_progress_callback;
+ void *parallel_progress_callback_arg;
} ParallelContext;
typedef struct ParallelWorkerContext
--
2.32.1 (Apple Git-133)
[application/octet-stream] v11-0002-Show-progress-for-index-vacuums.patch (9.5K, ../[email protected]/3-v11-0002-Show-progress-for-index-vacuums.patch)
download | inline diff:
From 5d0c7f7f761316d1089b010bc750cf36803b53e3 Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Thu, 26 May 2022 08:02:11 -0500
Subject: [PATCH v11 2/2] Show progress for index vacuums
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 25 ++++++++++++++++++
src/backend/access/heap/vacuumlazy.c | 19 ++++++++++++++
src/backend/catalog/system_views.sql | 3 ++-
src/backend/commands/vacuumparallel.c | 37 +++++++++++++++++++++++++++
src/include/commands/progress.h | 2 ++
src/test/regress/expected/rules.out | 4 ++-
6 files changed, 88 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 56d9b37..0599384 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6396,6 +6396,31 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when there are no indexes to process
+ or when failsafe is triggered during the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_completed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes already processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index b802ed2..f01341e 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -418,6 +418,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+ /* Report the number of indexes to vacuum/cleanup */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2327,6 +2330,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /* Report the number of indexes vacuumed */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
if (lazy_check_wraparound_failsafe(vacrel))
{
@@ -2361,6 +2366,10 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
vacrel->dead_items->num_items == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
+ /* Report that we're done vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
/*
* Increase and report the number of index scans.
*
@@ -2625,6 +2634,10 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ /* Report that we're no longer vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
get_database_name(MyDatabaseId),
@@ -2671,6 +2684,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+ /* Report the number of indexes cleaned */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
}
}
else
@@ -2680,6 +2695,10 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->num_index_scans,
estimated_count);
}
+
+ /* Report that we're done cleaning up indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index fedaed5..0c8261b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1163,7 +1163,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_completed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 1753da6..ad61a54 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -30,6 +30,7 @@
#include "access/table.h"
#include "access/xact.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -213,6 +214,7 @@ static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation
static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_index_scans,
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
+static void parallel_vacuum_progress_callback(void *arg);
/*
* Try to enter parallel mode and create a parallel context. Then initialize
@@ -288,6 +290,10 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
+ /* Setup the Parallel Progress Callback */
+ pvs->pcxt->parallel_progress_callback = parallel_vacuum_progress_callback;
+ pvs->pcxt->parallel_progress_callback_arg = pvs;
+
/*
* Estimate space for BufferUsage and WalUsage --
* PARALLEL_VACUUM_KEY_BUFFER_USAGE and PARALLEL_VACUUM_KEY_WAL_USAGE.
@@ -885,6 +891,13 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+
+ /*
+ * If we are the leader, update index vacuum progress.
+ */
+ if (!IsParallelWorker())
+ pvs->pcxt->parallel_progress_callback(pvs->pcxt->parallel_progress_callback_arg);
+
pfree(pvs->indname);
pvs->indname = NULL;
}
@@ -1071,3 +1084,27 @@ parallel_vacuum_error_callback(void *arg)
return;
}
}
+
+/*
+ * Callback to report index vacuum progress.
+ * Vacuum Progress should only be reported
+ * to the leader process.
+ */
+static void
+parallel_vacuum_progress_callback(void *arg)
+{
+ ParallelVacuumState *pvs = (ParallelVacuumState *)arg;
+ int indexes_completed = 0;
+
+ Assert(!IsParallelWorker());
+
+ for (int i = 0; i < pvs->nindexes; i++)
+ {
+ PVIndStats *indstats = &(pvs->indstats[i]);
+
+ if (indstats->status == PARALLEL_INDVAC_STATUS_COMPLETED)
+ indexes_completed++;
+ }
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed);
+}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938c..0d1724e 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_INDEXES_TOTAL 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index fc3cde3..9c3442f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2017,7 +2017,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_completed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT s.stats_reset,
--
2.32.1 (Apple Git-133)
view thread (96+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Add index scan progress to pg_stat_progress_vacuum
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox