public inbox for [email protected]
help / color / mirror / Atom feedRe: Add index scan progress to pg_stat_progress_vacuum
74+ messages / 11 participants
[nested] [flat]
* Re: Add index scan progress to pg_stat_progress_vacuum
@ 2022-01-11 01:01 Imseih (AWS), Sami <[email protected]>
2022-01-11 19:01 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-01-11 01:01 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
I agree, Renaming "index_vacuum_count" can be taken up in a separate discussion.
I have attached the 3rd revision of the patch which also includes the documentation changes. Also attached is a rendered html of the docs for review.
"max_index_vacuum_cycle_time" has been removed.
"index_rows_vacuumed" renamed to "index_tuples_removed". "tuples" is a more consistent with the terminology used.
"vacuum_cycle_ordinal_position" renamed to "index_ordinal_position".
On 1/10/22, 12:30 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/6/22, 6:14 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
> I am hesitant to make column name changes for obvious reasons, as it breaks existing tooling. However, I think there is a really good case to change "index_vacuum_count" as the name is confusing. "index_vacuum_cycles_completed" is the name I suggest if we agree to rename.
>
> For the new column, "num_indexes_to_vacuum" is good with me.
Yeah, I think we can skip renaming index_vacuum_count for now. In any
case, it would probably be good to discuss that in a separate thread.
Nathan
Attachments:
[text/html] progress-reporting.html (50.1K, ../../[email protected]/2-progress-reporting.html)
download | inline:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>28.4. Progress Reporting</title><link rel="stylesheet" type="text/css" href="https://www.postgresql.org/media/css/docs-complete.css" /><link rev="made" href="[email protected]" /><meta name="generator" content="DocBook XSL Stylesheets V1.79.1" /><link rel="prev" href="monitoring-locks.html" title="28.3. Viewing Locks" /><link rel="next" href="dynamic-trace.html" title="28.5. Dynamic Tracing" /></head><body id="docContent" class="container-fluid col-10"><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="5" align="center">28.4. Progress Reporting</th></tr><tr><td width="10%" align="left"><a accesskey="p" href="monitoring-locks.html" title="28.3. Viewing Locks">Prev</a> </td><td width="10%" align="left"><a accesskey="u" href="monitoring.html" title="Chapter 28. Monitoring Database Activity">Up</a></td><th width="60%" align="center">Chapter 28. Monitoring Database Activity</th><td width="10%" align="right"><a accesskey="h" href="index.html" title="PostgreSQL 15devel Documentation">Home</a></td><td width="10%" align="right"> <a accesskey="n" href="dynamic-trace.html" title="28.5. Dynamic Tracing">Next</a></td></tr></table><hr></hr></div><div class="sect1" id="PROGRESS-REPORTING"><div class="titlepage"><div><div><h2 class="title" style="clear: both">28.4. Progress Reporting</h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="sect2"><a href="progress-reporting.html#ANALYZE-PROGRESS-REPORTING">28.4.1. ANALYZE Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING">28.4.2. CREATE INDEX Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#VACUUM-PROGRESS-REPORTING">28.4.3. VACUUM Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#CLUSTER-PROGRESS-REPORTING">28.4.4. CLUSTER Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#BASEBACKUP-PROGRESS-REPORTING">28.4.5. Base Backup Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#COPY-PROGRESS-REPORTING">28.4.6. COPY Progress Reporting</a></span></dt></dl></div><p>
<span class="productname">PostgreSQL</span> has the ability to report the progress of
certain commands during command execution. Currently, the only commands
which support progress reporting are <code class="command">ANALYZE</code>,
<code class="command">CLUSTER</code>,
<code class="command">CREATE INDEX</code>, <code class="command">VACUUM</code>,
<code class="command">COPY</code>,
and <a class="xref" href="protocol-replication.html#PROTOCOL-REPLICATION-BASE-BACKUP">BASE_BACKUP</a> (i.e., replication
command that <a class="xref" href="app-pgbasebackup.html" title="pg_basebackup"><span class="refentrytitle"><span class="application">pg_basebackup</span></span></a> issues to take
a base backup).
This may be expanded in the future.
</p><div class="sect2" id="ANALYZE-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.1. ANALYZE Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.3.2" class="indexterm"></a><p>
Whenever <code class="command">ANALYZE</code> is running, the
<code class="structname">pg_stat_progress_analyze</code> view will contain a
row for each backend that is currently running that command. The tables
below describe the information that will be reported and provide
information about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-ANALYZE-VIEW"><p class="title"><strong>Table 28.35. <code class="structname">pg_stat_progress_analyze</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_analyze View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table being analyzed.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase. See <a class="xref" href="progress-reporting.html#ANALYZE-PHASES" title="Table 28.36. ANALYZE phases">Table 28.36</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">sample_blks_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of heap blocks that will be sampled.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">sample_blks_scanned</code> <code class="type">bigint</code>
</p>
<p>
Number of heap blocks scanned.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">ext_stats_total</code> <code class="type">bigint</code>
</p>
<p>
Number of extended statistics.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">ext_stats_computed</code> <code class="type">bigint</code>
</p>
<p>
Number of extended statistics computed. This counter only advances
when the phase is <code class="literal">computing extended statistics</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">child_tables_total</code> <code class="type">bigint</code>
</p>
<p>
Number of child tables.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">child_tables_done</code> <code class="type">bigint</code>
</p>
<p>
Number of child tables scanned. This counter only advances when the
phase is <code class="literal">acquiring inherited sample rows</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">current_child_table_relid</code> <code class="type">oid</code>
</p>
<p>
OID of the child table currently being scanned. This field is
only valid when the phase is
<code class="literal">acquiring inherited sample rows</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="ANALYZE-PHASES"><p class="title"><strong>Table 28.36. ANALYZE phases</strong></p><div class="table-contents"><table class="table" summary="ANALYZE phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
The command is preparing to begin scanning the heap. This phase is
expected to be very brief.
</td></tr><tr><td><code class="literal">acquiring sample rows</code></td><td>
The command is currently scanning the table given by
<code class="structfield">relid</code> to obtain sample rows.
</td></tr><tr><td><code class="literal">acquiring inherited sample rows</code></td><td>
The command is currently scanning child tables to obtain sample rows.
Columns <code class="structfield">child_tables_total</code>,
<code class="structfield">child_tables_done</code>, and
<code class="structfield">current_child_table_relid</code> contain the
progress information for this phase.
</td></tr><tr><td><code class="literal">computing statistics</code></td><td>
The command is computing statistics from the sample rows obtained
during the table scan.
</td></tr><tr><td><code class="literal">computing extended statistics</code></td><td>
The command is computing extended statistics from the sample rows
obtained during the table scan.
</td></tr><tr><td><code class="literal">finalizing analyze</code></td><td>
The command is updating <code class="structname">pg_class</code>. When this
phase is completed, <code class="command">ANALYZE</code> will end.
</td></tr></tbody></table></div></div><br class="table-break" /><div class="note"><h3 class="title">Note</h3><p>
Note that when <code class="command">ANALYZE</code> is run on a partitioned table,
all of its partitions are also recursively analyzed.
In that case, <code class="command">ANALYZE</code>
progress is reported first for the parent table, whereby its inheritance
statistics are collected, followed by that for each partition.
</p></div></div><div class="sect2" id="CREATE-INDEX-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.2. CREATE INDEX Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.4.2" class="indexterm"></a><p>
Whenever <code class="command">CREATE INDEX</code> or <code class="command">REINDEX</code> is running, the
<code class="structname">pg_stat_progress_create_index</code> view will contain
one row for each backend that is currently creating indexes. The tables
below describe the information that will be reported and provide information
about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-CREATE-INDEX-VIEW"><p class="title"><strong>Table 28.37. <code class="structname">pg_stat_progress_create_index</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_create_index View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table on which the index is being created.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_relid</code> <code class="type">oid</code>
</p>
<p>
OID of the index being created or reindexed. During a
non-concurrent <code class="command">CREATE INDEX</code>, this is 0.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">command</code> <code class="type">text</code>
</p>
<p>
The command that is running: <code class="literal">CREATE INDEX</code>,
<code class="literal">CREATE INDEX CONCURRENTLY</code>,
<code class="literal">REINDEX</code>, or <code class="literal">REINDEX CONCURRENTLY</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase of index creation. See <a class="xref" href="progress-reporting.html#CREATE-INDEX-PHASES" title="Table 28.38. CREATE INDEX Phases">Table 28.38</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">lockers_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of lockers to wait for, when applicable.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">lockers_done</code> <code class="type">bigint</code>
</p>
<p>
Number of lockers already waited for.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">current_locker_pid</code> <code class="type">bigint</code>
</p>
<p>
Process ID of the locker currently being waited for.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">blocks_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of blocks to be processed in the current phase.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">blocks_done</code> <code class="type">bigint</code>
</p>
<p>
Number of blocks already processed in the current phase.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of tuples to be processed in the current phase.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_done</code> <code class="type">bigint</code>
</p>
<p>
Number of tuples already processed in the current phase.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">partitions_total</code> <code class="type">bigint</code>
</p>
<p>
When creating an index on a partitioned table, this column is set to
the total number of partitions on which the index is to be created.
This field is <code class="literal">0</code> during a <code class="literal">REINDEX</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">partitions_done</code> <code class="type">bigint</code>
</p>
<p>
When creating an index on a partitioned table, this column is set to
the number of partitions on which the index has been created.
This field is <code class="literal">0</code> during a <code class="literal">REINDEX</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="CREATE-INDEX-PHASES"><p class="title"><strong>Table 28.38. CREATE INDEX Phases</strong></p><div class="table-contents"><table class="table" summary="CREATE INDEX Phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
<code class="command">CREATE INDEX</code> or <code class="command">REINDEX</code> is preparing to create the index. This
phase is expected to be very brief.
</td></tr><tr><td><code class="literal">waiting for writers before build</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> or <code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
with write locks that can potentially see the table to finish.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr><tr><td><code class="literal">building index</code></td><td>
The index is being built by the access method-specific code. In this phase,
access methods that support progress reporting fill in their own progress data,
and the subphase is indicated in this column. Typically,
<code class="structname">blocks_total</code> and <code class="structname">blocks_done</code>
will contain progress data, as well as potentially
<code class="structname">tuples_total</code> and <code class="structname">tuples_done</code>.
</td></tr><tr><td><code class="literal">waiting for writers before validation</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> or <code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
with write locks that can potentially write into the table to finish.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr><tr><td><code class="literal">index validation: scanning index</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> is scanning the index searching
for tuples that need to be validated.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">blocks_total</code> (set to the total size of the index)
and <code class="structname">blocks_done</code> contain the progress information for this phase.
</td></tr><tr><td><code class="literal">index validation: sorting tuples</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> is sorting the output of the
index scanning phase.
</td></tr><tr><td><code class="literal">index validation: scanning table</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> is scanning the table
to validate the index tuples collected in the previous two phases.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">blocks_total</code> (set to the total size of the table)
and <code class="structname">blocks_done</code> contain the progress information for this phase.
</td></tr><tr><td><code class="literal">waiting for old snapshots</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> or <code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
that can potentially see the table to release their snapshots. This
phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr><tr><td><code class="literal">waiting for readers before marking dead</code></td><td>
<code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
with read locks on the table to finish, before marking the old index dead.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr><tr><td><code class="literal">waiting for readers before dropping</code></td><td>
<code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
with read locks on the table to finish, before dropping the old index.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="sect2" id="VACUUM-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.3. VACUUM Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.5.2" class="indexterm"></a><p>
Whenever <code class="command">VACUUM</code> is running, the
<code class="structname">pg_stat_progress_vacuum</code> view will contain
one row for each backend (including autovacuum worker processes) that is
currently vacuuming. The <code class="structname">pg_stat_progress_vacuum_index</code>
view will also contain one row for each backend that is currently vacuuming an index.
The tables below describe the information that will be reported and provide
information about how to interpret it.
Progress for <code class="command">VACUUM FULL</code> commands is reported via
<code class="structname">pg_stat_progress_cluster</code>
because both <code class="command">VACUUM FULL</code> and <code class="command">CLUSTER</code>
rewrite the table, while regular <code class="command">VACUUM</code> only modifies it
in place. See <a class="xref" href="progress-reporting.html#CLUSTER-PROGRESS-REPORTING" title="28.4.4. CLUSTER Progress Reporting">Section 28.4.4</a>.
</p><div class="table" id="PG-STAT-PROGRESS-VACUUM-VIEW"><p class="title"><strong>Table 28.39. <code class="structname">pg_stat_progress_vacuum</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_vacuum View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table being vacuumed.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase of vacuum. See <a class="xref" href="progress-reporting.html#VACUUM-PHASES" title="Table 28.41. VACUUM Phases">Table 28.41</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of heap blocks in the table. This number is reported
as of the beginning of the scan; blocks added later will not be (and
need not be) visited by this <code class="command">VACUUM</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_scanned</code> <code class="type">bigint</code>
</p>
<p>
Number of heap blocks scanned. Because the
<a class="link" href="storage-vm.html" title="70.4. Visibility Map">visibility map</a> is used to optimize scans,
some blocks will be skipped without inspection; skipped blocks are
included in this total, so that this number will eventually become
equal to <code class="structfield">heap_blks_total</code> when the vacuum is complete.
This counter only advances when the phase is <code class="literal">scanning heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_vacuumed</code> <code class="type">bigint</code>
</p>
<p>
Number of heap blocks vacuumed. Unless the table has no indexes, this
counter only advances when the phase is <code class="literal">vacuuming heap</code>.
Blocks that contain no dead tuples are skipped, so the counter may
sometimes skip forward in large increments.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_vacuum_count</code> <code class="type">bigint</code>
</p>
<p>
Number of completed index vacuum cycles.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">max_dead_tuples</code> <code class="type">bigint</code>
</p>
<p>
Number of dead tuples that we can store before needing to perform
an index vacuum cycle, based on
<a class="xref" href="runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM">maintenance_work_mem</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">num_dead_tuples</code> <code class="type">bigint</code>
</p>
<p>
Number of dead tuples collected since the last index vacuum cycle.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">num_indexes_to_vacuum</code> <code class="type">bigint</code>
</p>
<p>
The number of indexes that will be vacuumed. Only indexes with
<code class="literal">pg_index.indisready</code> set to "true" will be vacuumed.
Whenever <a class="xref" href="runtime-config-client.html#GUC-VACUUM-FAILSAFE-AGE">vacuum_failsafe_age</a> is triggered, index
vacuuming will be bypassed.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="PG-STAT-PROGRESS-VACUUM-INDEX-VIEW"><p class="title"><strong>Table 28.40. <code class="structname">pg_stat_progress_vacuum_index</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_vacuum_index View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table being vacuumed.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">leader_pid</code> <code class="type">bigint</code>
</p>
<p>
Process ID of the parallel group leader. This field is <code class="literal">NULL</code>
if this process is a parallel group leader or the
<code class="literal">vacuuming indexes</code> phase is not performed in parallel.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">indexrelid</code> <code class="type">bigint</code>
</p>
<p>
OID of the index being vacuumed.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_ordinal_position</code> <code class="type">bigint</code>
</p>
<p>
The order in which the index is being vacuumed. Indexes are vacuumed by OID in ascending order.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_tuples_removed</code> <code class="type">bigint</code>
</p>
<p>
The number of tuple indexes removed during the <code class="literal">vacuuming indexes</code> phase of <code class="literal">VACUUM</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="VACUUM-PHASES"><p class="title"><strong>Table 28.41. VACUUM Phases</strong></p><div class="table-contents"><table class="table" summary="VACUUM Phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
<code class="command">VACUUM</code> is preparing to begin scanning the heap. This
phase is expected to be very brief.
</td></tr><tr><td><code class="literal">scanning heap</code></td><td>
<code class="command">VACUUM</code> is currently scanning the heap. It will prune and
defragment each page if required, and possibly perform freezing
activity. The <code class="structfield">heap_blks_scanned</code> column can be used
to monitor the progress of the scan.
</td></tr><tr><td><code class="literal">vacuuming indexes</code></td><td>
<code class="command">VACUUM</code> is currently vacuuming the indexes. If a table has
any indexes, this will happen at least once per vacuum, after the heap
has been completely scanned. It may happen multiple times per vacuum
if <a class="xref" href="runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM">maintenance_work_mem</a> (or, in the case of autovacuum,
<a class="xref" href="runtime-config-resource.html#GUC-AUTOVACUUM-WORK-MEM">autovacuum_work_mem</a> if set) is insufficient to store
the number of dead tuples found.
</td></tr><tr><td><code class="literal">vacuuming heap</code></td><td>
<code class="command">VACUUM</code> is currently vacuuming the heap. Vacuuming the heap
is distinct from scanning the heap, and occurs after each instance of
vacuuming indexes. If <code class="structfield">heap_blks_scanned</code> is less than
<code class="structfield">heap_blks_total</code>, the system will return to scanning
the heap after this phase is completed; otherwise, it will begin
cleaning up indexes after this phase is completed.
</td></tr><tr><td><code class="literal">cleaning up indexes</code></td><td>
<code class="command">VACUUM</code> is currently cleaning up indexes. This occurs after
the heap has been completely scanned and all vacuuming of the indexes
and the heap has been completed.
</td></tr><tr><td><code class="literal">truncating heap</code></td><td>
<code class="command">VACUUM</code> is currently truncating the heap so as to return
empty pages at the end of the relation to the operating system. This
occurs after cleaning up indexes.
</td></tr><tr><td><code class="literal">performing final cleanup</code></td><td>
<code class="command">VACUUM</code> is performing final cleanup. During this phase,
<code class="command">VACUUM</code> will vacuum the free space map, update statistics
in <code class="literal">pg_class</code>, and report statistics to the statistics
collector. When this phase is completed, <code class="command">VACUUM</code> will end.
</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="sect2" id="CLUSTER-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.4. CLUSTER Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.6.2" class="indexterm"></a><p>
Whenever <code class="command">CLUSTER</code> or <code class="command">VACUUM FULL</code> is
running, the <code class="structname">pg_stat_progress_cluster</code> view will
contain a row for each backend that is currently running either command.
The tables below describe the information that will be reported and
provide information about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-CLUSTER-VIEW"><p class="title"><strong>Table 28.42. <code class="structname">pg_stat_progress_cluster</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_cluster View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table being clustered.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">command</code> <code class="type">text</code>
</p>
<p>
The command that is running. Either <code class="literal">CLUSTER</code> or <code class="literal">VACUUM FULL</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase. See <a class="xref" href="progress-reporting.html#CLUSTER-PHASES" title="Table 28.43. CLUSTER and VACUUM FULL Phases">Table 28.43</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">cluster_index_relid</code> <code class="type">oid</code>
</p>
<p>
If the table is being scanned using an index, this is the OID of the
index being used; otherwise, it is zero.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_tuples_scanned</code> <code class="type">bigint</code>
</p>
<p>
Number of heap tuples scanned.
This counter only advances when the phase is
<code class="literal">seq scanning heap</code>,
<code class="literal">index scanning heap</code>
or <code class="literal">writing new heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_tuples_written</code> <code class="type">bigint</code>
</p>
<p>
Number of heap tuples written.
This counter only advances when the phase is
<code class="literal">seq scanning heap</code>,
<code class="literal">index scanning heap</code>
or <code class="literal">writing new heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of heap blocks in the table. This number is reported
as of the beginning of <code class="literal">seq scanning heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_scanned</code> <code class="type">bigint</code>
</p>
<p>
Number of heap blocks scanned. This counter only advances when the
phase is <code class="literal">seq scanning heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_rebuild_count</code> <code class="type">bigint</code>
</p>
<p>
Number of indexes rebuilt. This counter only advances when the phase
is <code class="literal">rebuilding index</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="CLUSTER-PHASES"><p class="title"><strong>Table 28.43. CLUSTER and VACUUM FULL Phases</strong></p><div class="table-contents"><table class="table" summary="CLUSTER and VACUUM FULL Phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
The command is preparing to begin scanning the heap. This phase is
expected to be very brief.
</td></tr><tr><td><code class="literal">seq scanning heap</code></td><td>
The command is currently scanning the table using a sequential scan.
</td></tr><tr><td><code class="literal">index scanning heap</code></td><td>
<code class="command">CLUSTER</code> is currently scanning the table using an index scan.
</td></tr><tr><td><code class="literal">sorting tuples</code></td><td>
<code class="command">CLUSTER</code> is currently sorting tuples.
</td></tr><tr><td><code class="literal">writing new heap</code></td><td>
<code class="command">CLUSTER</code> is currently writing the new heap.
</td></tr><tr><td><code class="literal">swapping relation files</code></td><td>
The command is currently swapping newly-built files into place.
</td></tr><tr><td><code class="literal">rebuilding index</code></td><td>
The command is currently rebuilding an index.
</td></tr><tr><td><code class="literal">performing final cleanup</code></td><td>
The command is performing final cleanup. When this phase is
completed, <code class="command">CLUSTER</code>
or <code class="command">VACUUM FULL</code> will end.
</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="sect2" id="BASEBACKUP-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.5. Base Backup Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.7.2" class="indexterm"></a><p>
Whenever an application like <span class="application">pg_basebackup</span>
is taking a base backup, the
<code class="structname">pg_stat_progress_basebackup</code>
view will contain a row for each WAL sender process that is currently
running the <code class="command">BASE_BACKUP</code> replication command
and streaming the backup. The tables below describe the information
that will be reported and provide information about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-BASEBACKUP-VIEW"><p class="title"><strong>Table 28.44. <code class="structname">pg_stat_progress_basebackup</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_basebackup View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of a WAL sender process.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase. See <a class="xref" href="progress-reporting.html#BASEBACKUP-PHASES" title="Table 28.45. Base backup phases">Table 28.45</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">backup_total</code> <code class="type">bigint</code>
</p>
<p>
Total amount of data that will be streamed. This is estimated and
reported as of the beginning of
<code class="literal">streaming database files</code> phase. Note that
this is only an approximation since the database
may change during <code class="literal">streaming database files</code> phase
and WAL log may be included in the backup later. This is always
the same value as <code class="structfield">backup_streamed</code>
once the amount of data streamed exceeds the estimated
total size. If the estimation is disabled in
<span class="application">pg_basebackup</span>
(i.e., <code class="literal">--no-estimate-size</code> option is specified),
this is <code class="literal">NULL</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">backup_streamed</code> <code class="type">bigint</code>
</p>
<p>
Amount of data streamed. This counter only advances
when the phase is <code class="literal">streaming database files</code> or
<code class="literal">transferring wal files</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tablespaces_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of tablespaces that will be streamed.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tablespaces_streamed</code> <code class="type">bigint</code>
</p>
<p>
Number of tablespaces streamed. This counter only
advances when the phase is <code class="literal">streaming database files</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="BASEBACKUP-PHASES"><p class="title"><strong>Table 28.45. Base backup phases</strong></p><div class="table-contents"><table class="table" summary="Base backup phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
The WAL sender process is preparing to begin the backup.
This phase is expected to be very brief.
</td></tr><tr><td><code class="literal">waiting for checkpoint to finish</code></td><td>
The WAL sender process is currently performing
<code class="function">pg_start_backup</code> to prepare to
take a base backup, and waiting for the start-of-backup
checkpoint to finish.
</td></tr><tr><td><code class="literal">estimating backup size</code></td><td>
The WAL sender process is currently estimating the total amount
of database files that will be streamed as a base backup.
</td></tr><tr><td><code class="literal">streaming database files</code></td><td>
The WAL sender process is currently streaming database files
as a base backup.
</td></tr><tr><td><code class="literal">waiting for wal archiving to finish</code></td><td>
The WAL sender process is currently performing
<code class="function">pg_stop_backup</code> to finish the backup,
and waiting for all the WAL files required for the base backup
to be successfully archived.
If either <code class="literal">--wal-method=none</code> or
<code class="literal">--wal-method=stream</code> is specified in
<span class="application">pg_basebackup</span>, the backup will end
when this phase is completed.
</td></tr><tr><td><code class="literal">transferring wal files</code></td><td>
The WAL sender process is currently transferring all WAL logs
generated during the backup. This phase occurs after
<code class="literal">waiting for wal archiving to finish</code> phase if
<code class="literal">--wal-method=fetch</code> is specified in
<span class="application">pg_basebackup</span>. The backup will end
when this phase is completed.
</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="sect2" id="COPY-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.6. COPY Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.8.2" class="indexterm"></a><p>
Whenever <code class="command">COPY</code> is running, the
<code class="structname">pg_stat_progress_copy</code> view will contain one row
for each backend that is currently running a <code class="command">COPY</code> command.
The table below describes the information that will be reported and provides
information about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-COPY-VIEW"><p class="title"><strong>Table 28.46. <code class="structname">pg_stat_progress_copy</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_copy View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table on which the <code class="command">COPY</code> command is
executed. It is set to <code class="literal">0</code> if copying from a
<code class="command">SELECT</code> query.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">command</code> <code class="type">text</code>
</p>
<p>
The command that is running: <code class="literal">COPY FROM</code>, or
<code class="literal">COPY TO</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">type</code> <code class="type">text</code>
</p>
<p>
The io type that the data is read from or written to:
<code class="literal">FILE</code>, <code class="literal">PROGRAM</code>,
<code class="literal">PIPE</code> (for <code class="command">COPY FROM STDIN</code> and
<code class="command">COPY TO STDOUT</code>), or <code class="literal">CALLBACK</code>
(used for example during the initial table synchronization in
logical replication).
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">bytes_processed</code> <code class="type">bigint</code>
</p>
<p>
Number of bytes already processed by <code class="command">COPY</code> command.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">bytes_total</code> <code class="type">bigint</code>
</p>
<p>
Size of source file for <code class="command">COPY FROM</code> command in bytes.
It is set to <code class="literal">0</code> if not available.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_processed</code> <code class="type">bigint</code>
</p>
<p>
Number of tuples already processed by <code class="command">COPY</code> command.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_excluded</code> <code class="type">bigint</code>
</p>
<p>
Number of tuples not processed because they were excluded by the
<code class="command">WHERE</code> clause of the <code class="command">COPY</code> command.
</p></td></tr></tbody></table></div></div><br class="table-break" /></div></div><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navfooter"><hr></hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="monitoring-locks.html" title="28.3. Viewing Locks">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="monitoring.html" title="Chapter 28. Monitoring Database Activity">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="dynamic-trace.html" title="28.5. Dynamic Tracing">Next</a></td></tr><tr><td width="40%" align="left" valign="top">28.3. Viewing Locks </td><td width="20%" align="center"><a accesskey="h" href="index.html" title="PostgreSQL 15devel Documentation">Home</a></td><td width="40%" align="right" valign="top"> 28.5. Dynamic Tracing</td></tr></table></div></body></html>
[application/octet-stream] 0001-Expose-progress-for-the-vacuuming-indexes-phase-of-a.patch (20.0K, ../../[email protected]/3-0001-Expose-progress-for-the-vacuuming-indexes-phase-of-a.patch)
download | inline diff:
From 61c5dd12727c94491131a53e15566c475b5d47f2 Mon Sep 17 00:00:00 2001
From: "Imseih (AWS), Sami" <[email protected]>
Date: Mon, 10 Jan 2022 04:13:03 +0000
Subject: [PATCH 1/1] Expose progress for the "vacuuming indexes" phase of a
VACUUM operation.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/5478DFCD-2333-401A-B2F0-0D186AB09228%40amazon.com
---
doc/src/sgml/monitoring.sgml | 110 +++++++++++++++++++++++++-
src/backend/access/gin/ginvacuum.c | 4 +
src/backend/access/gist/gistvacuum.c | 4 +
src/backend/access/hash/hash.c | 2 +
src/backend/access/heap/vacuumlazy.c | 19 ++++-
src/backend/access/nbtree/nbtree.c | 2 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 17 +++-
src/backend/commands/vacuumparallel.c | 22 +++++-
src/include/commands/progress.h | 19 +++--
src/test/regress/expected/rules.out | 20 ++++-
11 files changed, 205 insertions(+), 18 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 62f2a3332b..e82e2753bf 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6135,8 +6135,10 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Whenever <command>VACUUM</command> is running, the
<structname>pg_stat_progress_vacuum</structname> view will contain
one row for each backend (including autovacuum worker processes) that is
- currently vacuuming. The tables below describe the information
- that will be reported and provide information about how to interpret it.
+ currently vacuuming. The <structname>pg_stat_progress_vacuum_index</structname>
+ view will also contain one row for each backend that is currently vacuuming an index.
+ The tables below describe the information that will be reported and provide
+ information about how to interpret it.
Progress for <command>VACUUM FULL</command> commands is reported via
<structname>pg_stat_progress_cluster</structname>
because both <command>VACUUM FULL</command> and <command>CLUSTER</command>
@@ -6269,6 +6271,110 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>num_indexes_to_vacuum</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes that will be vacuumed. Only indexes with
+ <literal>pg_index.indisready</literal> set to "true" will be vacuumed.
+ Whenever <xref linkend="guc-vacuum-failsafe-age"/> is triggered, index
+ vacuuming will be bypassed.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>relid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the table being vacuumed.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Process ID of the parallel group leader. This field is <literal>NULL</literal>
+ if this process is a parallel group leader or the
+ <literal>vacuuming indexes</literal> phase is not performed in parallel.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>bigint</type>
+ </para>
+ <para>
+ OID of the index being vacuumed.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>index_ordinal_position</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The order in which the index is being vacuumed. Indexes are vacuumed by OID in ascending order.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>index_tuples_removed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of tuple indexes removed during the <literal>vacuuming indexes</literal> phase of <literal>VACUUM</literal>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..b7beeab751 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,8 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED, gvs->result->tuples_removed);
+
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..d288edc8cd 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,8 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED, vstate->stats->tuples_removed);
+
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index d48c8a4549..c1df85bce8 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -631,6 +631,8 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED, stats->tuples_removed);
+
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 5e4a5d0d82..c73d2013a0 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -343,6 +343,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, -1);
vacuum_set_xid_limits(rel,
params->freeze_min_age,
@@ -447,6 +448,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ /* Advertise the number of indexes to vacuum if we are not in failsafe mode */
+ if (!lazy_check_wraparound_failsafe(vacrel))
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEX_VACUUM, vacrel->nindexes);
/*
* Call lazy_scan_heap to perform all required heap pruning, index
* vacuuming, and heap vacuuming (plus related processing)
@@ -2038,6 +2042,7 @@ static bool
lazy_vacuum_all_indexes(LVRelState *vacrel)
{
bool allindexes = true;
+ PGRUsage ru0;
Assert(vacrel->nindexes > 0);
Assert(vacrel->do_index_vacuuming);
@@ -2056,6 +2061,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+ pg_rusage_init(&ru0);
+
if (!ParallelVacuumIsActive(vacrel))
{
for (int idx = 0; idx < vacrel->nindexes; idx++)
@@ -2063,14 +2070,22 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index that is being vacuumed and the vacuum order of the index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_CURRENT_INDRELID, indrel->rd_id);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_ORDINAL, idx + 1);
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /* Advertise that we are done vacuuming the index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_CURRENT_INDRELID, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
allindexes = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEX_VACUUM, 0);
break;
}
}
@@ -2085,8 +2100,10 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* Do a postcheck to consider applying wraparound failsafe now. Note
* that parallel VACUUM only gets the precheck and this postcheck.
*/
- if (lazy_check_wraparound_failsafe(vacrel))
+ if (lazy_check_wraparound_failsafe(vacrel)) {
allindexes = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEX_VACUUM, 0);
+ }
}
/*
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 13024af2fa..27cedf20b9 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1272,6 +1272,8 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED, stats->tuples_removed);
+
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..9753eb8090 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 701ff38f76..5bba12f678 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1124,9 +1124,22 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param11 AS num_indexes_to_vacuum
FROM pg_stat_get_progress_info('VACUUM') AS S
- LEFT JOIN pg_database D ON S.datid = D.oid;
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE (S.pid = S.param9 OR S.param9 = -1); -- show vacuum progress for the leader PID only
+
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid AS pid, S.datid AS datid, D.datname AS datname,
+ S.relid AS relid,
+ CASE S.param9 WHEN -1 THEN NULL ELSE S.param9 END AS leader_pid,
+ S.param8 AS indexrelid,
+ S.param10 AS index_ordinal_position, S.param12 AS index_tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param8 > 0; -- show vacuum progress for a PID that has advertised an index relid
CREATE VIEW pg_stat_progress_cluster AS
SELECT
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 53a59da2d6..6bb4a37d96 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -206,7 +207,7 @@ static void parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int nu
static void parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs);
static void parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs);
static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
- PVIndStats *indstats);
+ PVIndStats *indstats, int ordinal_position);
static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_index_scans,
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
@@ -754,7 +755,7 @@ parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs)
continue;
/* Do vacuum or cleanup of the index */
- parallel_vacuum_process_one_index(pvs, pvs->indrels[idx], indstats);
+ parallel_vacuum_process_one_index(pvs, pvs->indrels[idx], indstats, idx + 1);
}
/*
@@ -795,7 +796,7 @@ parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
continue;
/* Do vacuum or cleanup of the index */
- parallel_vacuum_process_one_index(pvs, pvs->indrels[i], indstats);
+ parallel_vacuum_process_one_index(pvs, pvs->indrels[i], indstats, i + 1);
}
/*
@@ -814,7 +815,7 @@ parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
*/
static void
parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
- PVIndStats *indstats)
+ PVIndStats *indstats, int ordinal_position)
{
IndexBulkDeleteResult *istat = NULL;
IndexBulkDeleteResult *istat_res;
@@ -842,7 +843,14 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ /* Advertise the index that is being vacuumed and the vacuum order of the index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_CURRENT_INDRELID, indrel->rd_id);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_ORDINAL, ordinal_position);
+
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+
+ /* Advertise that we are done vacuuming the index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_CURRENT_INDRELID, 0);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
@@ -942,6 +950,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
int nindexes;
char *sharedquery;
ErrorContextCallback errcallback;
+ PGPROC *leader = MyProc->lockGroupLeader;
/*
* A parallel vacuum worker must have only PROC_IN_VACUUM flag since we
@@ -965,6 +974,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
+ RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, leader->pid);
+
/*
* Open all indexes. indrels are sorted in order by OID, which should be
* matched to the leader's one.
@@ -1035,6 +1048,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
FreeAccessStrategy(pvs.bstrategy);
+ pgstat_progress_end_command();
}
/*
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..fe3971f075 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -18,13 +18,18 @@
#define PROGRESS_H
/* Progress parameters for (lazy) vacuum */
-#define PROGRESS_VACUUM_PHASE 0
-#define PROGRESS_VACUUM_TOTAL_HEAP_BLKS 1
-#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
-#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
-#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_PHASE 0
+#define PROGRESS_VACUUM_TOTAL_HEAP_BLKS 1
+#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
+#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
+#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
+#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_CURRENT_INDRELID 7
+#define PROGRESS_VACUUM_LEADER_PID 8
+#define PROGRESS_VACUUM_INDEX_ORDINAL 9
+#define PROGRESS_VACUUM_TOTAL_INDEX_VACUUM 10
+#define PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index b58b062b10..a00e0db28c 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2024,9 +2024,25 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param11 AS num_indexes_to_vacuum
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
- LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE ((s.pid = s.param9) OR (s.param9 = '-1'::integer));
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.relid,
+ CASE s.param9
+ WHEN '-1'::integer THEN NULL::bigint
+ ELSE s.param9
+ END AS leader_pid,
+ s.param8 AS indexrelid,
+ s.param10 AS index_ordinal_position,
+ s.param12 AS index_tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE (s.param8 > 0);
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-01-11 19:01 ` Bossart, Nathan <[email protected]>
2022-01-11 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Bossart, Nathan @ 2022-01-11 19:01 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; Justin Pryzby <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
On 1/10/22, 5:01 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
> I have attached the 3rd revision of the patch which also includes the documentation changes. Also attached is a rendered html of the docs for review.
>
> "max_index_vacuum_cycle_time" has been removed.
> "index_rows_vacuumed" renamed to "index_tuples_removed". "tuples" is a more consistent with the terminology used.
> "vacuum_cycle_ordinal_position" renamed to "index_ordinal_position".
Thanks for the new version of the patch!
nitpick: I get one whitespace error when applying the patch.
Applying: Expose progress for the "vacuuming indexes" phase of a VACUUM operation.
.git/rebase-apply/patch:44: tab in indent.
Whenever <xref linkend="guc-vacuum-failsafe-age"/> is triggered, index
warning: 1 line adds whitespace errors.
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>num_indexes_to_vacuum</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes that will be vacuumed. Only indexes with
+ <literal>pg_index.indisready</literal> set to "true" will be vacuumed.
+ Whenever <xref linkend="guc-vacuum-failsafe-age"/> is triggered, index
+ vacuuming will be bypassed.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
We may want to avoid exhaustively listing the cases when this value
will be zero. I would suggest saying, "When index cleanup is skipped,
this value will be zero" instead.
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>relid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the table being vacuumed.
+ </para></entry>
+ </row>
Do we need to include this field? I would expect indexrelid to go
here.
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Process ID of the parallel group leader. This field is <literal>NULL</literal>
+ if this process is a parallel group leader or the
+ <literal>vacuuming indexes</literal> phase is not performed in parallel.
+ </para></entry>
+ </row>
Are there cases where the parallel group leader will have an entry in
this view when parallelism is enabled?
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>index_ordinal_position</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The order in which the index is being vacuumed. Indexes are vacuumed by OID in ascending order.
+ </para></entry>
+ </row>
Should we include the bit about the OID ordering? I suppose that is
unlikely to change in the near future, but I don't know if it is
relevant information. Also, do we need to include the "index_"
prefix? This view is specific for indexes. (I have the same question
for index_tuples_removed.)
Should this new table go after the "VACUUM phases" table? It might
make sense to keep the phases table closer to where it is referenced.
+ /* Advertise the number of indexes to vacuum if we are not in failsafe mode */
+ if (!lazy_check_wraparound_failsafe(vacrel))
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEX_VACUUM, vacrel->nindexes);
Shouldn't this be 0 when INDEX_CLEANUP is off, too?
+#define PROGRESS_VACUUM_CURRENT_INDRELID 7
+#define PROGRESS_VACUUM_LEADER_PID 8
+#define PROGRESS_VACUUM_INDEX_ORDINAL 9
+#define PROGRESS_VACUUM_TOTAL_INDEX_VACUUM 10
+#define PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED 11
nitpick: I would suggest the following names to match the existing
style:
PROGRESS_VACUUM_NUM_INDEXES_TO_VACUUM
PROGRESS_VACUUM_INDEX_LEADER_PID
PROGRESS_VACUUM_INDEX_INDEXRELID
PROGRESS_VACUUM_INDEX_ORDINAL_POSITION
PROGRESS_VACUUM_INDEX_TUPLES_REMOVED
Nathan
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-11 19:01 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
@ 2022-01-11 20:33 ` Imseih (AWS), Sami <[email protected]>
2022-01-11 22:18 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-01-11 20:33 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
On 1/11/22, 1:01 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/10/22, 5:01 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
> I have attached the 3rd revision of the patch which also includes the documentation changes. Also attached is a rendered html of the docs for review.
>
> "max_index_vacuum_cycle_time" has been removed.
> "index_rows_vacuumed" renamed to "index_tuples_removed". "tuples" is a more consistent with the terminology used.
> "vacuum_cycle_ordinal_position" renamed to "index_ordinal_position".
Thanks for the new version of the patch!
nitpick: I get one whitespace error when applying the patch.
Applying: Expose progress for the "vacuuming indexes" phase of a VACUUM operation.
.git/rebase-apply/patch:44: tab in indent.
Whenever <xref linkend="guc-vacuum-failsafe-age"/> is triggered, index
warning: 1 line adds whitespace errors.
That was missed. Will fix it.
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>num_indexes_to_vacuum</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes that will be vacuumed. Only indexes with
+ <literal>pg_index.indisready</literal> set to "true" will be vacuumed.
+ Whenever <xref linkend="guc-vacuum-failsafe-age"/> is triggered, index
+ vacuuming will be bypassed.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
We may want to avoid exhaustively listing the cases when this value
will be zero. I would suggest saying, "When index cleanup is skipped,
this value will be zero" instead.
What about something like "The number of indexes that are eligible for vacuuming".
This covers the cases where either an individual index is skipped or the entire "index vacuuming" phase is skipped.
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>relid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the table being vacuumed.
+ </para></entry>
+ </row>
Do we need to include this field? I would expect indexrelid to go
here.
Having indexrelid and relid makes the pg_stat_progress_vacuum_index view "self-contained". A user can lookup the index and table being vacuumed without joining back to pg_stat_progress_vacuum.
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Process ID of the parallel group leader. This field is <literal>NULL</literal>
+ if this process is a parallel group leader or the
+ <literal>vacuuming indexes</literal> phase is not performed in parallel.
+ </para></entry>
+ </row>
Are there cases where the parallel group leader will have an entry in
this view when parallelism is enabled?
Yes. A parallel group leader can perform an index vacuum just like a parallel worker. If you do something like "vacuum (parallel 3) ", you may have up to 4 processes vacuuming indexes. The leader + 3 workers.
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>index_ordinal_position</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The order in which the index is being vacuumed. Indexes are vacuumed by OID in ascending order.
+ </para></entry>
+ </row>
Should we include the bit about the OID ordering? I suppose that is
unlikely to change in the near future, but I don't know if it is
relevant information. Also, do we need to include the "index_"
prefix? This view is specific for indexes. (I have the same question
for index_tuples_removed.)
I was on the fence about both of these as well. Will make a change to this.
Should this new table go after the "VACUUM phases" table? It might
make sense to keep the phases table closer to where it is referenced.
I did not think that would read better. The introduction discusses both views and the "phase" table is linked from the pg_stat_progress_vacuum
+ /* Advertise the number of indexes to vacuum if we are not in failsafe mode */
+ if (!lazy_check_wraparound_failsafe(vacrel))
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEX_VACUUM, vacrel->nindexes);
Shouldn't this be 0 when INDEX_CLEANUP is off, too?
This view is only covering the "vacuum index" phase, but it should also cover index_cleanup phase as well. Will update the patch.
+#define PROGRESS_VACUUM_CURRENT_INDRELID 7
+#define PROGRESS_VACUUM_LEADER_PID 8
+#define PROGRESS_VACUUM_INDEX_ORDINAL 9
+#define PROGRESS_VACUUM_TOTAL_INDEX_VACUUM 10
+#define PROGRESS_VACUUM_DEAD_TUPLES_VACUUMED 11
nitpick: I would suggest the following names to match the existing
style:
PROGRESS_VACUUM_NUM_INDEXES_TO_VACUUM
PROGRESS_VACUUM_INDEX_LEADER_PID
PROGRESS_VACUUM_INDEX_INDEXRELID
PROGRESS_VACUUM_INDEX_ORDINAL_POSITION
PROGRESS_VACUUM_INDEX_TUPLES_REMOVED
That looks better.
Nathan
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-11 19:01 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-11 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-01-11 22:18 ` Bossart, Nathan <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Bossart, Nathan @ 2022-01-11 22:18 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; Justin Pryzby <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
On 1/11/22, 12:33 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
> What about something like "The number of indexes that are eligible for vacuuming".
> This covers the cases where either an individual index is skipped or the entire "index vacuuming" phase is skipped.
Hm. I don't know if "eligible" is the right word. An index can be
eligible for vacuuming but skipped because we set INDEX_CLEANUP to
false. Maybe we should just stick with "The number of indexes that
will be vacuumed." The only thing we may want to clarify is whether
this value will change in some cases (e.g., vacuum failsafe takes
effect).
Nathan
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-01-12 07:44 ` Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-01-12 07:44 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
(We had better avoid top-posting[1])
On Tue, Jan 11, 2022 at 10:01 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> I agree, Renaming "index_vacuum_count" can be taken up in a separate discussion.
>
> I have attached the 3rd revision of the patch which also includes the documentation changes. Also attached is a rendered html of the docs for review.
Thank you for updating the patch!
Regarding the new pg_stat_progress_vacuum_index view, why do we need
to have a separate view? Users will have to check two views. If this
view is expected to be used together with and joined to
pg_stat_progress_vacuum, why don't we provide one view that has full
information from the beginning? Especially, I think it's not useful
that the total number of indexes to vacuum (num_indexes_to_vacuum
column) and the current number of indexes that have been vacuumed
(index_ordinal_position column) are shown in separate views.
Also, I’m not sure how useful index_tuples_removed is; what can we
infer from this value (without a total number)?
Regards,
[1] https://en.wikipedia.org/wiki/Posting_style#Top-posting
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-01-12 19:28 ` Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Bossart, Nathan @ 2022-01-12 19:28 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; Imseih (AWS), Sami <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
I suppose we could add all of the new columns to
pg_stat_progress_vacuum and just set columns to NULL as appropriate.
But is that really better than having a separate view?
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
I think the idea was that you can compare it against max_dead_tuples
and num_dead_tuples to get an estimate of the current cycle progress.
Otherwise, it just shows that progress is being made.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
@ 2022-01-13 03:52 ` Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-01-13 03:52 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
On 1/12/22, 1:28 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
> I suppose we could add all of the new columns to
> pg_stat_progress_vacuum and just set columns to NULL as appropriate.
> But is that really better than having a separate view?
To add, since a vacuum can utilize parallel worker processes + the main vacuum process to perform index vacuuming, it made sense to separate the backends doing index vacuum/cleanup in a separate view.
Besides what Nathan suggested, the only other clean option I can think of is to perhaps create a json column in pg_stat_progress_vacuum which will include all the new fields. My concern with this approach is that it will make usability, to flatten the json, difficult for users.
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
> I think the idea was that you can compare it against max_dead_tuples
> and num_dead_tuples to get an estimate of the current cycle progress.
> Otherwise, it just shows that progress is being made.
The main purpose is to really show that the "index vacuum" phase is actually making progress. Note that for certain types of indexes, i.e. GIN/GIST the number of tuples_removed will end up exceeding the number of num_dead_tuples.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-01-27 02:07 ` Imseih (AWS), Sami <[email protected]>
2022-01-27 02:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 21:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 3 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-01-27 02:07 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
Attached is the latest patch and associated documentation.
This version addresses the index_ordinal_position column confusion. Rather than displaying the index position, the pg_stat_progress_vacuum view now has 2 new column(s):
index_total - this column will show the total number of indexes to be vacuumed
index_complete_count - this column will show the total number of indexes processed so far. In order to deal with the parallel vacuums, the parallel_workers ( planned workers ) value had to be exposed and each backends performing an index vacuum/cleanup in parallel had to advertise the number of indexes it vacuumed/cleaned. The # of indexes vacuumed for the parallel cleanup can then be derived the pg_stat_progress_vacuum view.
postgres=# \d pg_stat_progress_vacuum
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default
----------------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
relid | oid | | |
phase | text | | |
heap_blks_total | bigint | | |
heap_blks_scanned | bigint | | |
heap_blks_vacuumed | bigint | | |
index_vacuum_count | bigint | | |
max_dead_tuples | bigint | | |
num_dead_tuples | bigint | | |
index_total | bigint | | |. <<<---------------------
index_complete_count | numeric | | |. <<<---------------------
The pg_stat_progress_vacuum_index view includes:
Indexrelid - the currently vacuumed index
Leader_pid - the pid of the leader process. NULL if the process is the leader or vacuum is not parallel
tuples_removed - the amount of indexes tuples removed. The user can use this column to see that the index vacuum has movement.
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
phase | text | | |
leader_pid | bigint | | |
tuples_removed | bigint | | |
On 1/12/22, 9:52 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
On 1/12/22, 1:28 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
> I suppose we could add all of the new columns to
> pg_stat_progress_vacuum and just set columns to NULL as appropriate.
> But is that really better than having a separate view?
To add, since a vacuum can utilize parallel worker processes + the main vacuum process to perform index vacuuming, it made sense to separate the backends doing index vacuum/cleanup in a separate view.
Besides what Nathan suggested, the only other clean option I can think of is to perhaps create a json column in pg_stat_progress_vacuum which will include all the new fields. My concern with this approach is that it will make usability, to flatten the json, difficult for users.
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
> I think the idea was that you can compare it against max_dead_tuples
> and num_dead_tuples to get an estimate of the current cycle progress.
> Otherwise, it just shows that progress is being made.
The main purpose is to really show that the "index vacuum" phase is actually making progress. Note that for certain types of indexes, i.e. GIN/GIST the number of tuples_removed will end up exceeding the number of num_dead_tuples.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
Attachments:
[text/html] monitoring.html (9.1K, ../../[email protected]/2-monitoring.html)
download | inline:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Chapter 28. Monitoring Database Activity</title><link rel="stylesheet" type="text/css" href="https://www.postgresql.org/media/css/docs-complete.css" /><link rev="made" href="[email protected]" /><meta name="generator" content="DocBook XSL Stylesheets V1.79.1" /><link rel="prev" href="hot-standby.html" title="27.4. Hot Standby" /><link rel="next" href="monitoring-ps.html" title="28.1. Standard Unix Tools" /></head><body id="docContent" class="container-fluid col-10"><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="5" align="center">Chapter 28. Monitoring Database Activity</th></tr><tr><td width="10%" align="left"><a accesskey="p" href="hot-standby.html" title="27.4. Hot Standby">Prev</a> </td><td width="10%" align="left"><a accesskey="u" href="admin.html" title="Part III. Server Administration">Up</a></td><th width="60%" align="center">Part III. Server Administration</th><td width="10%" align="right"><a accesskey="h" href="index.html" title="PostgreSQL 15devel Documentation">Home</a></td><td width="10%" align="right"> <a accesskey="n" href="monitoring-ps.html" title="28.1. Standard Unix Tools">Next</a></td></tr></table><hr></hr></div><div class="chapter" id="MONITORING"><div class="titlepage"><div><div><h2 class="title">Chapter 28. Monitoring Database Activity</h2></div></div></div><div class="toc"><p><strong>Table of Contents</strong></p><dl class="toc"><dt><span class="sect1"><a href="monitoring-ps.html">28.1. Standard Unix Tools</a></span></dt><dt><span class="sect1"><a href="monitoring-stats.html">28.2. The Statistics Collector</a></span></dt><dd><dl><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-STATS-SETUP">28.2.1. Statistics Collection Configuration</a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-STATS-VIEWS">28.2.2. Viewing Statistics</a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW">28.2.3. <code class="structname">pg_stat_activity</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-REPLICATION-VIEW">28.2.4. <code class="structname">pg_stat_replication</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-REPLICATION-SLOTS-VIEW">28.2.5. <code class="structname">pg_stat_replication_slots</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-WAL-RECEIVER-VIEW">28.2.6. <code class="structname">pg_stat_wal_receiver</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-SUBSCRIPTION">28.2.7. <code class="structname">pg_stat_subscription</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-SUBSCRIPTION-WORKERS">28.2.8. <code class="structname">pg_stat_subscription_workers</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-SSL-VIEW">28.2.9. <code class="structname">pg_stat_ssl</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-GSSAPI-VIEW">28.2.10. <code class="structname">pg_stat_gssapi</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-ARCHIVER-VIEW">28.2.11. <code class="structname">pg_stat_archiver</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-BGWRITER-VIEW">28.2.12. <code class="structname">pg_stat_bgwriter</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-WAL-VIEW">28.2.13. <code class="structname">pg_stat_wal</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-DATABASE-VIEW">28.2.14. <code class="structname">pg_stat_database</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-DATABASE-CONFLICTS-VIEW">28.2.15. <code class="structname">pg_stat_database_conflicts</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-ALL-TABLES-VIEW">28.2.16. <code class="structname">pg_stat_all_tables</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-ALL-INDEXES-VIEW">28.2.17. <code class="structname">pg_stat_all_indexes</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STATIO-ALL-TABLES-VIEW">28.2.18. <code class="structname">pg_statio_all_tables</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STATIO-ALL-INDEXES-VIEW">28.2.19. <code class="structname">pg_statio_all_indexes</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STATIO-ALL-SEQUENCES-VIEW">28.2.20. <code class="structname">pg_statio_all_sequences</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-USER-FUNCTIONS-VIEW">28.2.21. <code class="structname">pg_stat_user_functions</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-PG-STAT-SLRU-VIEW">28.2.22. <code class="structname">pg_stat_slru</code></a></span></dt><dt><span class="sect2"><a href="monitoring-stats.html#MONITORING-STATS-FUNCTIONS">28.2.23. Statistics Functions</a></span></dt></dl></dd><dt><span class="sect1"><a href="monitoring-locks.html">28.3. Viewing Locks</a></span></dt><dt><span class="sect1"><a href="progress-reporting.html">28.4. Progress Reporting</a></span></dt><dd><dl><dt><span class="sect2"><a href="progress-reporting.html#ANALYZE-PROGRESS-REPORTING">28.4.1. ANALYZE Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING">28.4.2. CREATE INDEX Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#VACUUM-PROGRESS-REPORTING">28.4.3. VACUUM Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#CLUSTER-PROGRESS-REPORTING">28.4.4. CLUSTER Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#BASEBACKUP-PROGRESS-REPORTING">28.4.5. Base Backup Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#COPY-PROGRESS-REPORTING">28.4.6. COPY Progress Reporting</a></span></dt></dl></dd><dt><span class="sect1"><a href="dynamic-trace.html">28.5. Dynamic Tracing</a></span></dt><dd><dl><dt><span class="sect2"><a href="dynamic-trace.html#COMPILING-FOR-TRACE">28.5.1. Compiling for Dynamic Tracing</a></span></dt><dt><span class="sect2"><a href="dynamic-trace.html#TRACE-POINTS">28.5.2. Built-in Probes</a></span></dt><dt><span class="sect2"><a href="dynamic-trace.html#USING-TRACE-POINTS">28.5.3. Using Probes</a></span></dt><dt><span class="sect2"><a href="dynamic-trace.html#DEFINING-TRACE-POINTS">28.5.4. Defining New Probes</a></span></dt></dl></dd></dl></div><a id="id-1.6.15.2" class="indexterm"></a><a id="id-1.6.15.3" class="indexterm"></a><p>
A database administrator frequently wonders, <span class="quote">“<span class="quote">What is the system
doing right now?</span>”</span>
This chapter discusses how to find that out.
</p><p>
Several tools are available for monitoring database activity and
analyzing performance. Most of this chapter is devoted to describing
<span class="productname">PostgreSQL</span>'s statistics collector,
but one should not neglect regular Unix monitoring programs such as
<code class="command">ps</code>, <code class="command">top</code>, <code class="command">iostat</code>, and <code class="command">vmstat</code>.
Also, once one has identified a
poorly-performing query, further investigation might be needed using
<span class="productname">PostgreSQL</span>'s <a class="link" href="sql-explain.html" title="EXPLAIN"><code class="command">EXPLAIN</code></a> command.
<a class="xref" href="using-explain.html" title="14.1. Using EXPLAIN">Section 14.1</a> discusses <code class="command">EXPLAIN</code>
and other methods for understanding the behavior of an individual
query.
</p></div><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navfooter"><hr></hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="hot-standby.html" title="27.4. Hot Standby">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="admin.html" title="Part III. Server Administration">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="monitoring-ps.html" title="28.1. Standard Unix Tools">Next</a></td></tr><tr><td width="40%" align="left" valign="top">27.4. Hot Standby </td><td width="20%" align="center"><a accesskey="h" href="index.html" title="PostgreSQL 15devel Documentation">Home</a></td><td width="40%" align="right" valign="top"> 28.1. Standard Unix Tools</td></tr></table></div></body></html>
[application/octet-stream] 0004-Expose-progress-for-the-vacuuming-indexes-cleanup-ph.patch (28.5K, ../../[email protected]/3-0004-Expose-progress-for-the-vacuuming-indexes-cleanup-ph.patch)
download | inline diff:
From 33d4116a6b2ea4eef7d00ada279a329854514068 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)"
<[email protected]>
Date: Thu, 27 Jan 2022 01:41:20 +0000
Subject: [PATCH 1/1] Expose progress for the "vacuuming indexes/cleanup" phase
of a VACUUM operation.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/5478DFCD-2333-401A-B2F0-0D186AB09228%40amazon.com
---
doc/src/sgml/monitoring.sgml | 131 ++++++++++++++++++++++++++
src/backend/access/gin/ginvacuum.c | 4 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 11 +++
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 70 ++++++++++----
src/backend/commands/vacuumparallel.c | 56 ++++++++---
src/include/commands/progress.h | 6 ++
src/test/regress/expected/rules.out | 63 ++++++++++---
11 files changed, 309 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 62f2a3332b..b2b33acb66 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -379,6 +379,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_progress_vacuum_index</structname><indexterm><primary>pg_stat_progress_vacuum_index</primary></indexterm></entry>
+ <entry>One row for each backend (including autovacuum worker processes) running
+ the <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase of a
+ <command>VACUUM</command>, showing current progress.
+ See <xref linkend='vacuum-progress-reporting'/>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_cluster</structname><indexterm><primary>pg_stat_progress_cluster</primary></indexterm></entry>
<entry>One row for each backend running
@@ -6269,6 +6278,128 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>index_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The total number of indexes that will be processed by the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>index_complete_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The total number of indexes processed by the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> of the vacuum.
+ For vacuums with mutiple index vacuum cycles, this value will be set to <literal>0</literal>
+ at the start of each cycle.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..73660719b5 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,8 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
+
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index d48c8a4549..84341aa7e7 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -631,6 +631,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1749cc2a47..432229adea 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -340,6 +340,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
vacuum_set_xid_limits(rel,
params->freeze_min_age,
@@ -455,6 +456,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->FreezeLimit = FreezeLimit;
vacrel->MultiXactCutoff = MultiXactCutoff;
+ /* Advertise the total indexes to vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
+
/*
* Call lazy_scan_heap to perform all required heap pruning, index
* vacuuming, and heap vacuuming (plus related processing)
@@ -2010,9 +2014,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_COMPLETED, idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
if (lazy_check_wraparound_failsafe(vacrel))
{
@@ -2419,9 +2427,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_COMPLETED, idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
}
}
else
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 13024af2fa..7455d73d13 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1272,6 +1272,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3cb69b1f87..159605cb3f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1113,22 +1113,60 @@ CREATE VIEW pg_stat_progress_analyze AS
LEFT JOIN pg_database D ON S.datid = D.oid;
CREATE VIEW pg_stat_progress_vacuum AS
- SELECT
- S.pid AS pid, S.datid AS datid, D.datname AS datname,
- S.relid AS relid,
- CASE S.param1 WHEN 0 THEN 'initializing'
- WHEN 1 THEN 'scanning heap'
- WHEN 2 THEN 'vacuuming indexes'
- WHEN 3 THEN 'vacuuming heap'
- WHEN 4 THEN 'cleaning up indexes'
- WHEN 5 THEN 'truncating heap'
- WHEN 6 THEN 'performing final cleanup'
- END AS phase,
- S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
- S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
- FROM pg_stat_get_progress_info('VACUUM') AS S
- LEFT JOIN pg_database D ON S.datid = D.oid;
+ SELECT
+ S.pid,
+ S.datid,
+ S.datname,
+ S.relid,
+ S.phase,
+ S.heap_blks_total,
+ S.heap_blks_scanned,
+ S.heap_blks_vacuumed,
+ S.index_vacuum_count,
+ S.max_dead_tuples,
+ S.num_dead_tuples,
+ S.index_total,
+ S.index_complete_count
+ FROM (SELECT
+ S.pid AS pid, S.datid AS datid, D.datname AS datname,
+ S.relid AS relid, S.param12 as leader_pid,
+ CASE S.param1 WHEN 0 THEN 'initializing'
+ WHEN 1 THEN 'scanning heap'
+ WHEN 2 THEN 'vacuuming indexes'
+ WHEN 3 THEN 'vacuuming heap'
+ WHEN 4 THEN 'cleaning up indexes'
+ WHEN 5 THEN 'truncating heap'
+ WHEN 6 THEN 'performing final cleanup'
+ END AS phase,
+ S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
+ S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS index_total,
+ CASE WHEN S.param1 IN (2,4) AND S.param9 > 0
+ AND S.param10 > 0
+ AND S.param10 > (COUNT(*) OVER (PARTITION BY S.param12) - 1)
+ THEN S.param8 - (COUNT(*) OVER (PARTITION BY S.param12) - 1)
+ ELSE SUM(S.param9) OVER(PARTITION BY S.param12)
+ END AS index_complete_count
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid) AS S
+ WHERE S.pid = S.leader_pid;
+
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid AS pid, S.datid AS datid,
+ D.datname AS datname,
+ S.param11 AS indexrelid,
+ CASE S.param1 WHEN 2 THEN 'vacuuming indexes'
+ WHEN 4 THEN 'cleaning up indexes'
+ END AS phase,
+ CASE WHEN S.param12 = s.pid THEN NULL
+ ELSE S.param12
+ END AS leader_pid,
+ S.param13 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid =D.oid
+ WHERE S.param1 IN (2, 4) AND S.param11 NOT IN (0);
CREATE VIEW pg_stat_progress_cluster AS
SELECT
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..eb4becf900 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,12 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID */
+ int leader_pid;
+
+ /* Number of planned parallel workers */
+ int parallel_workers;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -204,10 +211,10 @@ static int parallel_vacuum_compute_workers(Relation *indrels, int nindexes, int
bool *will_parallel_vacuum);
static void parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scans,
bool vacuum);
-static void parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs);
-static void parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs);
-static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
- PVIndStats *indstats);
+static void parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs, int *idx_vacuumed);
+static void parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs, int *idx_vacuumed);
+static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
+ PVIndStats *indstats, int *idx_vacuumed);
static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_index_scans,
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
@@ -259,6 +266,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
return NULL;
}
+ pgstat_progress_update_param(PROGRESS_VACUUM_WORKERS_PLANNED, parallel_workers);
+
pvs = (ParallelVacuumState *) palloc0(sizeof(ParallelVacuumState));
pvs->indrels = indrels;
pvs->nindexes = nindexes;
@@ -357,6 +366,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->parallel_workers = parallel_workers;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -565,6 +576,7 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
{
int nworkers;
PVIndVacStatus new_status;
+ int idx_vacuumed = 0;
Assert(!IsParallelWorker());
@@ -671,13 +683,13 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
}
/* Vacuum the indexes that can be processed by only leader process */
- parallel_vacuum_process_unsafe_indexes(pvs);
+ parallel_vacuum_process_unsafe_indexes(pvs, &idx_vacuumed);
/*
* Join as a parallel worker. The leader vacuums alone processes all
* parallel-safe indexes in the case where no workers are launched.
*/
- parallel_vacuum_process_safe_indexes(pvs);
+ parallel_vacuum_process_safe_indexes(pvs, &idx_vacuumed);
/*
* Next, accumulate buffer and WAL usage. (This must wait for the workers
@@ -723,7 +735,7 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
* vacuum worker processes to vacuum the indexes in parallel.
*/
static void
-parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs)
+parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs, int *idx_vacuumed)
{
/*
* Increment the active worker count if we are able to launch any worker.
@@ -755,7 +767,8 @@ parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs)
continue;
/* Do vacuum or cleanup of the index */
- parallel_vacuum_process_one_index(pvs, pvs->indrels[idx], indstats);
+ parallel_vacuum_process_one_index(pvs, pvs->indrels[idx], indstats, idx_vacuumed);
+
}
/*
@@ -777,7 +790,7 @@ parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs)
* enforced by parallel_vacuum_compute_workers().
*/
static void
-parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
+parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs, int *idx_vacuumed)
{
Assert(!IsParallelWorker());
@@ -796,7 +809,8 @@ parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
continue;
/* Do vacuum or cleanup of the index */
- parallel_vacuum_process_one_index(pvs, pvs->indrels[i], indstats);
+ parallel_vacuum_process_one_index(pvs, pvs->indrels[i], indstats, idx_vacuumed);
+
}
/*
@@ -805,6 +819,7 @@ parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
*/
if (VacuumActiveNWorkers)
pg_atomic_sub_fetch_u32(VacuumActiveNWorkers, 1);
+
}
/*
@@ -815,7 +830,7 @@ parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
*/
static void
parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
- PVIndStats *indstats)
+ PVIndStats *indstats, int *idx_vacuumed)
{
IndexBulkDeleteResult *istat = NULL;
IndexBulkDeleteResult *istat_res;
@@ -840,13 +855,19 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ (*idx_vacuumed)++;
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ (*idx_vacuumed)++;
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
@@ -885,6 +906,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
pvs->indname = NULL;
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_COMPLETED, *idx_vacuumed);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
}
/*
@@ -943,6 +968,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
int nindexes;
char *sharedquery;
ErrorContextCallback errcallback;
+ int idx_vacuumed = 0;
/*
* A parallel vacuum worker must have only PROC_IN_VACUUM flag since we
@@ -965,6 +991,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM, RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
+ pgstat_progress_update_param(PROGRESS_VACUUM_WORKERS_PLANNED, shared->parallel_workers);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -973,6 +1002,8 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_open_indexes(rel, RowExclusiveLock, &nindexes, &indrels);
Assert(nindexes > 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, nindexes);
+
if (shared->maintenance_work_mem_worker > 0)
maintenance_work_mem = shared->maintenance_work_mem_worker;
@@ -1022,7 +1053,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrStartParallelQuery();
/* Process indexes to perform vacuum/cleanup */
- parallel_vacuum_process_safe_indexes(&pvs);
+ parallel_vacuum_process_safe_indexes(&pvs, &idx_vacuumed);
/* Report buffer/WAL usage during parallel execution */
buffer_usage = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_BUFFER_USAGE, false);
@@ -1036,6 +1067,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
FreeAccessStrategy(pvs.bstrategy);
+ pgstat_progress_end_command();
}
/*
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..83e74e44b3 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,12 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_NUM_INDEX_COMPLETED 8
+#define PROGRESS_VACUUM_WORKERS_PLANNED 9
+#define PROGRESS_VACUUM_INDEXRELID 10
+#define PROGRESS_VACUUM_LEADER_PID 11
+#define PROGRESS_VACUUM_TUPLES_REMOVED 12
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d652f7b5fb..ca2e31cd90 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2007,26 +2007,63 @@ pg_stat_progress_create_index| SELECT s.pid,
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_vacuum| SELECT s.pid,
s.datid,
- d.datname,
+ s.datname,
s.relid,
+ s.phase,
+ s.heap_blks_total,
+ s.heap_blks_scanned,
+ s.heap_blks_vacuumed,
+ s.index_vacuum_count,
+ s.max_dead_tuples,
+ s.num_dead_tuples,
+ s.index_total,
+ s.index_complete_count
+ FROM ( SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.relid,
+ s_1.param12 AS leader_pid,
+ CASE s_1.param1
+ WHEN 0 THEN 'initializing'::text
+ WHEN 1 THEN 'scanning heap'::text
+ WHEN 2 THEN 'vacuuming indexes'::text
+ WHEN 3 THEN 'vacuuming heap'::text
+ WHEN 4 THEN 'cleaning up indexes'::text
+ WHEN 5 THEN 'truncating heap'::text
+ WHEN 6 THEN 'performing final cleanup'::text
+ ELSE NULL::text
+ END AS phase,
+ s_1.param2 AS heap_blks_total,
+ s_1.param3 AS heap_blks_scanned,
+ s_1.param4 AS heap_blks_vacuumed,
+ s_1.param5 AS index_vacuum_count,
+ s_1.param6 AS max_dead_tuples,
+ s_1.param7 AS num_dead_tuples,
+ s_1.param8 AS index_total,
+ CASE
+ WHEN ((s_1.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint])) AND (s_1.param9 > 0) AND (s_1.param10 > 0) AND (s_1.param10 > (count(*) OVER (PARTITION BY s_1.param12) - 1))) THEN ((s_1.param8 - (count(*) OVER (PARTITION BY s_1.param12) - 1)))::numeric
+ ELSE sum(s_1.param9) OVER (PARTITION BY s_1.param12)
+ END AS index_complete_count
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))) s
+ WHERE (s.pid = s.leader_pid);
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.param11 AS indexrelid,
CASE s.param1
- WHEN 0 THEN 'initializing'::text
- WHEN 1 THEN 'scanning heap'::text
WHEN 2 THEN 'vacuuming indexes'::text
- WHEN 3 THEN 'vacuuming heap'::text
WHEN 4 THEN 'cleaning up indexes'::text
- WHEN 5 THEN 'truncating heap'::text
- WHEN 6 THEN 'performing final cleanup'::text
ELSE NULL::text
END AS phase,
- s.param2 AS heap_blks_total,
- s.param3 AS heap_blks_scanned,
- s.param4 AS heap_blks_vacuumed,
- s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ CASE
+ WHEN (s.param12 = s.pid) THEN NULL::bigint
+ ELSE s.param12
+ END AS leader_pid,
+ s.param13 AS tuples_removed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
- LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE ((s.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint])) AND (s.param11 <> 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-01-27 02:15 ` Imseih (AWS), Sami <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-01-27 02:15 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
My apologies. The last attachment of documentation was the wrong file. Attached is the correct documentation file.
Thanks
On 1/26/22, 8:07 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
Attached is the latest patch and associated documentation.
This version addresses the index_ordinal_position column confusion. Rather than displaying the index position, the pg_stat_progress_vacuum view now has 2 new column(s):
index_total - this column will show the total number of indexes to be vacuumed
index_complete_count - this column will show the total number of indexes processed so far. In order to deal with the parallel vacuums, the parallel_workers ( planned workers ) value had to be exposed and each backends performing an index vacuum/cleanup in parallel had to advertise the number of indexes it vacuumed/cleaned. The # of indexes vacuumed for the parallel cleanup can then be derived the pg_stat_progress_vacuum view.
postgres=# \d pg_stat_progress_vacuum
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default
----------------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
relid | oid | | |
phase | text | | |
heap_blks_total | bigint | | |
heap_blks_scanned | bigint | | |
heap_blks_vacuumed | bigint | | |
index_vacuum_count | bigint | | |
max_dead_tuples | bigint | | |
num_dead_tuples | bigint | | |
index_total | bigint | | |. <<<---------------------
index_complete_count | numeric | | |. <<<---------------------
The pg_stat_progress_vacuum_index view includes:
Indexrelid - the currently vacuumed index
Leader_pid - the pid of the leader process. NULL if the process is the leader or vacuum is not parallel
tuples_removed - the amount of indexes tuples removed. The user can use this column to see that the index vacuum has movement.
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
phase | text | | |
leader_pid | bigint | | |
tuples_removed | bigint | | |
On 1/12/22, 9:52 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
On 1/12/22, 1:28 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
> I suppose we could add all of the new columns to
> pg_stat_progress_vacuum and just set columns to NULL as appropriate.
> But is that really better than having a separate view?
To add, since a vacuum can utilize parallel worker processes + the main vacuum process to perform index vacuuming, it made sense to separate the backends doing index vacuum/cleanup in a separate view.
Besides what Nathan suggested, the only other clean option I can think of is to perhaps create a json column in pg_stat_progress_vacuum which will include all the new fields. My concern with this approach is that it will make usability, to flatten the json, difficult for users.
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
> I think the idea was that you can compare it against max_dead_tuples
> and num_dead_tuples to get an estimate of the current cycle progress.
> Otherwise, it just shows that progress is being made.
The main purpose is to really show that the "index vacuum" phase is actually making progress. Note that for certain types of indexes, i.e. GIN/GIST the number of tuples_removed will end up exceeding the number of num_dead_tuples.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
Attachments:
[text/html] progress-reporting.html (50.8K, ../../[email protected]/2-progress-reporting.html)
download | inline:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>28.4. Progress Reporting</title><link rel="stylesheet" type="text/css" href="https://www.postgresql.org/media/css/docs-complete.css" /><link rev="made" href="[email protected]" /><meta name="generator" content="DocBook XSL Stylesheets V1.79.1" /><link rel="prev" href="monitoring-locks.html" title="28.3. Viewing Locks" /><link rel="next" href="dynamic-trace.html" title="28.5. Dynamic Tracing" /></head><body id="docContent" class="container-fluid col-10"><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="5" align="center">28.4. Progress Reporting</th></tr><tr><td width="10%" align="left"><a accesskey="p" href="monitoring-locks.html" title="28.3. Viewing Locks">Prev</a> </td><td width="10%" align="left"><a accesskey="u" href="monitoring.html" title="Chapter 28. Monitoring Database Activity">Up</a></td><th width="60%" align="center">Chapter 28. Monitoring Database Activity</th><td width="10%" align="right"><a accesskey="h" href="index.html" title="PostgreSQL 15devel Documentation">Home</a></td><td width="10%" align="right"> <a accesskey="n" href="dynamic-trace.html" title="28.5. Dynamic Tracing">Next</a></td></tr></table><hr></hr></div><div class="sect1" id="PROGRESS-REPORTING"><div class="titlepage"><div><div><h2 class="title" style="clear: both">28.4. Progress Reporting</h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="sect2"><a href="progress-reporting.html#ANALYZE-PROGRESS-REPORTING">28.4.1. ANALYZE Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING">28.4.2. CREATE INDEX Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#VACUUM-PROGRESS-REPORTING">28.4.3. VACUUM Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#CLUSTER-PROGRESS-REPORTING">28.4.4. CLUSTER Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#BASEBACKUP-PROGRESS-REPORTING">28.4.5. Base Backup Progress Reporting</a></span></dt><dt><span class="sect2"><a href="progress-reporting.html#COPY-PROGRESS-REPORTING">28.4.6. COPY Progress Reporting</a></span></dt></dl></div><p>
<span class="productname">PostgreSQL</span> has the ability to report the progress of
certain commands during command execution. Currently, the only commands
which support progress reporting are <code class="command">ANALYZE</code>,
<code class="command">CLUSTER</code>,
<code class="command">CREATE INDEX</code>, <code class="command">VACUUM</code>,
<code class="command">COPY</code>,
and <a class="xref" href="protocol-replication.html#PROTOCOL-REPLICATION-BASE-BACKUP">BASE_BACKUP</a> (i.e., replication
command that <a class="xref" href="app-pgbasebackup.html" title="pg_basebackup"><span class="refentrytitle"><span class="application">pg_basebackup</span></span></a> issues to take
a base backup).
This may be expanded in the future.
</p><div class="sect2" id="ANALYZE-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.1. ANALYZE Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.3.2" class="indexterm"></a><p>
Whenever <code class="command">ANALYZE</code> is running, the
<code class="structname">pg_stat_progress_analyze</code> view will contain a
row for each backend that is currently running that command. The tables
below describe the information that will be reported and provide
information about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-ANALYZE-VIEW"><p class="title"><strong>Table 28.35. <code class="structname">pg_stat_progress_analyze</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_analyze View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table being analyzed.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase. See <a class="xref" href="progress-reporting.html#ANALYZE-PHASES" title="Table 28.36. ANALYZE phases">Table 28.36</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">sample_blks_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of heap blocks that will be sampled.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">sample_blks_scanned</code> <code class="type">bigint</code>
</p>
<p>
Number of heap blocks scanned.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">ext_stats_total</code> <code class="type">bigint</code>
</p>
<p>
Number of extended statistics.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">ext_stats_computed</code> <code class="type">bigint</code>
</p>
<p>
Number of extended statistics computed. This counter only advances
when the phase is <code class="literal">computing extended statistics</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">child_tables_total</code> <code class="type">bigint</code>
</p>
<p>
Number of child tables.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">child_tables_done</code> <code class="type">bigint</code>
</p>
<p>
Number of child tables scanned. This counter only advances when the
phase is <code class="literal">acquiring inherited sample rows</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">current_child_table_relid</code> <code class="type">oid</code>
</p>
<p>
OID of the child table currently being scanned. This field is
only valid when the phase is
<code class="literal">acquiring inherited sample rows</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="ANALYZE-PHASES"><p class="title"><strong>Table 28.36. ANALYZE phases</strong></p><div class="table-contents"><table class="table" summary="ANALYZE phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
The command is preparing to begin scanning the heap. This phase is
expected to be very brief.
</td></tr><tr><td><code class="literal">acquiring sample rows</code></td><td>
The command is currently scanning the table given by
<code class="structfield">relid</code> to obtain sample rows.
</td></tr><tr><td><code class="literal">acquiring inherited sample rows</code></td><td>
The command is currently scanning child tables to obtain sample rows.
Columns <code class="structfield">child_tables_total</code>,
<code class="structfield">child_tables_done</code>, and
<code class="structfield">current_child_table_relid</code> contain the
progress information for this phase.
</td></tr><tr><td><code class="literal">computing statistics</code></td><td>
The command is computing statistics from the sample rows obtained
during the table scan.
</td></tr><tr><td><code class="literal">computing extended statistics</code></td><td>
The command is computing extended statistics from the sample rows
obtained during the table scan.
</td></tr><tr><td><code class="literal">finalizing analyze</code></td><td>
The command is updating <code class="structname">pg_class</code>. When this
phase is completed, <code class="command">ANALYZE</code> will end.
</td></tr></tbody></table></div></div><br class="table-break" /><div class="note"><h3 class="title">Note</h3><p>
Note that when <code class="command">ANALYZE</code> is run on a partitioned table,
all of its partitions are also recursively analyzed.
In that case, <code class="command">ANALYZE</code>
progress is reported first for the parent table, whereby its inheritance
statistics are collected, followed by that for each partition.
</p></div></div><div class="sect2" id="CREATE-INDEX-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.2. CREATE INDEX Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.4.2" class="indexterm"></a><p>
Whenever <code class="command">CREATE INDEX</code> or <code class="command">REINDEX</code> is running, the
<code class="structname">pg_stat_progress_create_index</code> view will contain
one row for each backend that is currently creating indexes. The tables
below describe the information that will be reported and provide information
about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-CREATE-INDEX-VIEW"><p class="title"><strong>Table 28.37. <code class="structname">pg_stat_progress_create_index</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_create_index View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table on which the index is being created.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_relid</code> <code class="type">oid</code>
</p>
<p>
OID of the index being created or reindexed. During a
non-concurrent <code class="command">CREATE INDEX</code>, this is 0.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">command</code> <code class="type">text</code>
</p>
<p>
The command that is running: <code class="literal">CREATE INDEX</code>,
<code class="literal">CREATE INDEX CONCURRENTLY</code>,
<code class="literal">REINDEX</code>, or <code class="literal">REINDEX CONCURRENTLY</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase of index creation. See <a class="xref" href="progress-reporting.html#CREATE-INDEX-PHASES" title="Table 28.38. CREATE INDEX Phases">Table 28.38</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">lockers_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of lockers to wait for, when applicable.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">lockers_done</code> <code class="type">bigint</code>
</p>
<p>
Number of lockers already waited for.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">current_locker_pid</code> <code class="type">bigint</code>
</p>
<p>
Process ID of the locker currently being waited for.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">blocks_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of blocks to be processed in the current phase.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">blocks_done</code> <code class="type">bigint</code>
</p>
<p>
Number of blocks already processed in the current phase.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of tuples to be processed in the current phase.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_done</code> <code class="type">bigint</code>
</p>
<p>
Number of tuples already processed in the current phase.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">partitions_total</code> <code class="type">bigint</code>
</p>
<p>
When creating an index on a partitioned table, this column is set to
the total number of partitions on which the index is to be created.
This field is <code class="literal">0</code> during a <code class="literal">REINDEX</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">partitions_done</code> <code class="type">bigint</code>
</p>
<p>
When creating an index on a partitioned table, this column is set to
the number of partitions on which the index has been created.
This field is <code class="literal">0</code> during a <code class="literal">REINDEX</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="CREATE-INDEX-PHASES"><p class="title"><strong>Table 28.38. CREATE INDEX Phases</strong></p><div class="table-contents"><table class="table" summary="CREATE INDEX Phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
<code class="command">CREATE INDEX</code> or <code class="command">REINDEX</code> is preparing to create the index. This
phase is expected to be very brief.
</td></tr><tr><td><code class="literal">waiting for writers before build</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> or <code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
with write locks that can potentially see the table to finish.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr><tr><td><code class="literal">building index</code></td><td>
The index is being built by the access method-specific code. In this phase,
access methods that support progress reporting fill in their own progress data,
and the subphase is indicated in this column. Typically,
<code class="structname">blocks_total</code> and <code class="structname">blocks_done</code>
will contain progress data, as well as potentially
<code class="structname">tuples_total</code> and <code class="structname">tuples_done</code>.
</td></tr><tr><td><code class="literal">waiting for writers before validation</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> or <code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
with write locks that can potentially write into the table to finish.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr><tr><td><code class="literal">index validation: scanning index</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> is scanning the index searching
for tuples that need to be validated.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">blocks_total</code> (set to the total size of the index)
and <code class="structname">blocks_done</code> contain the progress information for this phase.
</td></tr><tr><td><code class="literal">index validation: sorting tuples</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> is sorting the output of the
index scanning phase.
</td></tr><tr><td><code class="literal">index validation: scanning table</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> is scanning the table
to validate the index tuples collected in the previous two phases.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">blocks_total</code> (set to the total size of the table)
and <code class="structname">blocks_done</code> contain the progress information for this phase.
</td></tr><tr><td><code class="literal">waiting for old snapshots</code></td><td>
<code class="command">CREATE INDEX CONCURRENTLY</code> or <code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
that can potentially see the table to release their snapshots. This
phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr><tr><td><code class="literal">waiting for readers before marking dead</code></td><td>
<code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
with read locks on the table to finish, before marking the old index dead.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr><tr><td><code class="literal">waiting for readers before dropping</code></td><td>
<code class="command">REINDEX CONCURRENTLY</code> is waiting for transactions
with read locks on the table to finish, before dropping the old index.
This phase is skipped when not in concurrent mode.
Columns <code class="structname">lockers_total</code>, <code class="structname">lockers_done</code>
and <code class="structname">current_locker_pid</code> contain the progress
information for this phase.
</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="sect2" id="VACUUM-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.3. VACUUM Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.5.2" class="indexterm"></a><p>
Whenever <code class="command">VACUUM</code> is running, the
<code class="structname">pg_stat_progress_vacuum</code> view will contain
one row for each backend (including autovacuum worker processes) that is
currently vacuuming. The tables below describe the information
that will be reported and provide information about how to interpret it.
Progress for <code class="command">VACUUM FULL</code> commands is reported via
<code class="structname">pg_stat_progress_cluster</code>
because both <code class="command">VACUUM FULL</code> and <code class="command">CLUSTER</code>
rewrite the table, while regular <code class="command">VACUUM</code> only modifies it
in place. See <a class="xref" href="progress-reporting.html#CLUSTER-PROGRESS-REPORTING" title="28.4.4. CLUSTER Progress Reporting">Section 28.4.4</a>.
</p><div class="table" id="PG-STAT-PROGRESS-VACUUM-VIEW"><p class="title"><strong>Table 28.39. <code class="structname">pg_stat_progress_vacuum</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_vacuum View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table being vacuumed.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase of vacuum. See <a class="xref" href="progress-reporting.html#VACUUM-PHASES" title="Table 28.41. VACUUM Phases">Table 28.41</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of heap blocks in the table. This number is reported
as of the beginning of the scan; blocks added later will not be (and
need not be) visited by this <code class="command">VACUUM</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_scanned</code> <code class="type">bigint</code>
</p>
<p>
Number of heap blocks scanned. Because the
<a class="link" href="storage-vm.html" title="70.4. Visibility Map">visibility map</a> is used to optimize scans,
some blocks will be skipped without inspection; skipped blocks are
included in this total, so that this number will eventually become
equal to <code class="structfield">heap_blks_total</code> when the vacuum is complete.
This counter only advances when the phase is <code class="literal">scanning heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_vacuumed</code> <code class="type">bigint</code>
</p>
<p>
Number of heap blocks vacuumed. Unless the table has no indexes, this
counter only advances when the phase is <code class="literal">vacuuming heap</code>.
Blocks that contain no dead tuples are skipped, so the counter may
sometimes skip forward in large increments.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_vacuum_count</code> <code class="type">bigint</code>
</p>
<p>
Number of completed index vacuum cycles.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">max_dead_tuples</code> <code class="type">bigint</code>
</p>
<p>
Number of dead tuples that we can store before needing to perform
an index vacuum cycle, based on
<a class="xref" href="runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM">maintenance_work_mem</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">num_dead_tuples</code> <code class="type">bigint</code>
</p>
<p>
Number of dead tuples collected since the last index vacuum cycle.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_total</code> <code class="type">bigint</code>
</p>
<p>
The total number of indexes that will be processed by the
<code class="literal">vacuuming indexes</code> or <code class="literal">cleaning up indexes</code> phase
of vacuum.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_complete_count</code> <code class="type">bigint</code>
</p>
<p>
The total number of indexes processed by the
<code class="literal">vacuuming indexes</code> or <code class="literal">cleaning up indexes</code> of the vacuum.
For vacuums with mutiple index vacuum cycles, this value will be set to <code class="literal">0</code>
at the start of each cycle.
</p></td></tr></tbody></table></div></div><br class="table-break" /><a id="id-1.6.15.9.5.5" class="indexterm"></a><p>
Whenever <code class="command">VACUUM</code> is running, the
<code class="structname">pg_stat_progress_vacuum_index</code> view will contain
one row for each backend (including autovacuum worker processes) that is
currently performing the <code class="literal">vacuuming indexes</code> or
<code class="literal">cleaning up indexes</code> phase of the vacuum.
</p><div class="table" id="PG-STAT-PROGRESS-VACUUM-INDEX-VIEW"><p class="title"><strong>Table 28.40. <code class="structname">pg_stat_progress_vacuum_index</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_vacuum_index View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">indexrelid</code> <code class="type">oid</code>
</p>
<p>
OID of the index being processed in the ongoing phase of the vacuum.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase of a vacuum. Only the
<code class="literal">vacuuming indexes</code> or <code class="literal">cleaning up indexes</code>
phase will be listed in this view. See <a class="xref" href="progress-reporting.html#VACUUM-PHASES" title="Table 28.41. VACUUM Phases">Table 28.41</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">leader_pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of the parallel vacuum leader, if this process is a
parallel vacuum worker. <code class="literal">NULL</code> if this process is a
parallel vacuum leader or does not participate in parallel vacuum.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_removed</code> <code class="type">oid</code>
</p>
<p>
The number of index tuples removed by the <code class="literal">vacuuming indexes</code> phase.
This field is <code class="literal">0</code> during the <code class="literal">cleaning up indexes</code>
phase.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="VACUUM-PHASES"><p class="title"><strong>Table 28.41. VACUUM Phases</strong></p><div class="table-contents"><table class="table" summary="VACUUM Phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
<code class="command">VACUUM</code> is preparing to begin scanning the heap. This
phase is expected to be very brief.
</td></tr><tr><td><code class="literal">scanning heap</code></td><td>
<code class="command">VACUUM</code> is currently scanning the heap. It will prune and
defragment each page if required, and possibly perform freezing
activity. The <code class="structfield">heap_blks_scanned</code> column can be used
to monitor the progress of the scan.
</td></tr><tr><td><code class="literal">vacuuming indexes</code></td><td>
<code class="command">VACUUM</code> is currently vacuuming the indexes. If a table has
any indexes, this will happen at least once per vacuum, after the heap
has been completely scanned. It may happen multiple times per vacuum
if <a class="xref" href="runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM">maintenance_work_mem</a> (or, in the case of autovacuum,
<a class="xref" href="runtime-config-resource.html#GUC-AUTOVACUUM-WORK-MEM">autovacuum_work_mem</a> if set) is insufficient to store
the number of dead tuples found.
</td></tr><tr><td><code class="literal">vacuuming heap</code></td><td>
<code class="command">VACUUM</code> is currently vacuuming the heap. Vacuuming the heap
is distinct from scanning the heap, and occurs after each instance of
vacuuming indexes. If <code class="structfield">heap_blks_scanned</code> is less than
<code class="structfield">heap_blks_total</code>, the system will return to scanning
the heap after this phase is completed; otherwise, it will begin
cleaning up indexes after this phase is completed.
</td></tr><tr><td><code class="literal">cleaning up indexes</code></td><td>
<code class="command">VACUUM</code> is currently cleaning up indexes. This occurs after
the heap has been completely scanned and all vacuuming of the indexes
and the heap has been completed.
</td></tr><tr><td><code class="literal">truncating heap</code></td><td>
<code class="command">VACUUM</code> is currently truncating the heap so as to return
empty pages at the end of the relation to the operating system. This
occurs after cleaning up indexes.
</td></tr><tr><td><code class="literal">performing final cleanup</code></td><td>
<code class="command">VACUUM</code> is performing final cleanup. During this phase,
<code class="command">VACUUM</code> will vacuum the free space map, update statistics
in <code class="literal">pg_class</code>, and report statistics to the statistics
collector. When this phase is completed, <code class="command">VACUUM</code> will end.
</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="sect2" id="CLUSTER-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.4. CLUSTER Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.6.2" class="indexterm"></a><p>
Whenever <code class="command">CLUSTER</code> or <code class="command">VACUUM FULL</code> is
running, the <code class="structname">pg_stat_progress_cluster</code> view will
contain a row for each backend that is currently running either command.
The tables below describe the information that will be reported and
provide information about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-CLUSTER-VIEW"><p class="title"><strong>Table 28.42. <code class="structname">pg_stat_progress_cluster</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_cluster View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table being clustered.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">command</code> <code class="type">text</code>
</p>
<p>
The command that is running. Either <code class="literal">CLUSTER</code> or <code class="literal">VACUUM FULL</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase. See <a class="xref" href="progress-reporting.html#CLUSTER-PHASES" title="Table 28.43. CLUSTER and VACUUM FULL Phases">Table 28.43</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">cluster_index_relid</code> <code class="type">oid</code>
</p>
<p>
If the table is being scanned using an index, this is the OID of the
index being used; otherwise, it is zero.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_tuples_scanned</code> <code class="type">bigint</code>
</p>
<p>
Number of heap tuples scanned.
This counter only advances when the phase is
<code class="literal">seq scanning heap</code>,
<code class="literal">index scanning heap</code>
or <code class="literal">writing new heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_tuples_written</code> <code class="type">bigint</code>
</p>
<p>
Number of heap tuples written.
This counter only advances when the phase is
<code class="literal">seq scanning heap</code>,
<code class="literal">index scanning heap</code>
or <code class="literal">writing new heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of heap blocks in the table. This number is reported
as of the beginning of <code class="literal">seq scanning heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">heap_blks_scanned</code> <code class="type">bigint</code>
</p>
<p>
Number of heap blocks scanned. This counter only advances when the
phase is <code class="literal">seq scanning heap</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">index_rebuild_count</code> <code class="type">bigint</code>
</p>
<p>
Number of indexes rebuilt. This counter only advances when the phase
is <code class="literal">rebuilding index</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="CLUSTER-PHASES"><p class="title"><strong>Table 28.43. CLUSTER and VACUUM FULL Phases</strong></p><div class="table-contents"><table class="table" summary="CLUSTER and VACUUM FULL Phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
The command is preparing to begin scanning the heap. This phase is
expected to be very brief.
</td></tr><tr><td><code class="literal">seq scanning heap</code></td><td>
The command is currently scanning the table using a sequential scan.
</td></tr><tr><td><code class="literal">index scanning heap</code></td><td>
<code class="command">CLUSTER</code> is currently scanning the table using an index scan.
</td></tr><tr><td><code class="literal">sorting tuples</code></td><td>
<code class="command">CLUSTER</code> is currently sorting tuples.
</td></tr><tr><td><code class="literal">writing new heap</code></td><td>
<code class="command">CLUSTER</code> is currently writing the new heap.
</td></tr><tr><td><code class="literal">swapping relation files</code></td><td>
The command is currently swapping newly-built files into place.
</td></tr><tr><td><code class="literal">rebuilding index</code></td><td>
The command is currently rebuilding an index.
</td></tr><tr><td><code class="literal">performing final cleanup</code></td><td>
The command is performing final cleanup. When this phase is
completed, <code class="command">CLUSTER</code>
or <code class="command">VACUUM FULL</code> will end.
</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="sect2" id="BASEBACKUP-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.5. Base Backup Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.7.2" class="indexterm"></a><p>
Whenever an application like <span class="application">pg_basebackup</span>
is taking a base backup, the
<code class="structname">pg_stat_progress_basebackup</code>
view will contain a row for each WAL sender process that is currently
running the <code class="command">BASE_BACKUP</code> replication command
and streaming the backup. The tables below describe the information
that will be reported and provide information about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-BASEBACKUP-VIEW"><p class="title"><strong>Table 28.44. <code class="structname">pg_stat_progress_basebackup</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_basebackup View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of a WAL sender process.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">phase</code> <code class="type">text</code>
</p>
<p>
Current processing phase. See <a class="xref" href="progress-reporting.html#BASEBACKUP-PHASES" title="Table 28.45. Base backup phases">Table 28.45</a>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">backup_total</code> <code class="type">bigint</code>
</p>
<p>
Total amount of data that will be streamed. This is estimated and
reported as of the beginning of
<code class="literal">streaming database files</code> phase. Note that
this is only an approximation since the database
may change during <code class="literal">streaming database files</code> phase
and WAL log may be included in the backup later. This is always
the same value as <code class="structfield">backup_streamed</code>
once the amount of data streamed exceeds the estimated
total size. If the estimation is disabled in
<span class="application">pg_basebackup</span>
(i.e., <code class="literal">--no-estimate-size</code> option is specified),
this is <code class="literal">NULL</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">backup_streamed</code> <code class="type">bigint</code>
</p>
<p>
Amount of data streamed. This counter only advances
when the phase is <code class="literal">streaming database files</code> or
<code class="literal">transferring wal files</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tablespaces_total</code> <code class="type">bigint</code>
</p>
<p>
Total number of tablespaces that will be streamed.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tablespaces_streamed</code> <code class="type">bigint</code>
</p>
<p>
Number of tablespaces streamed. This counter only
advances when the phase is <code class="literal">streaming database files</code>.
</p></td></tr></tbody></table></div></div><br class="table-break" /><div class="table" id="BASEBACKUP-PHASES"><p class="title"><strong>Table 28.45. Base backup phases</strong></p><div class="table-contents"><table class="table" summary="Base backup phases" border="1"><colgroup><col class="col1" /><col class="col2" /></colgroup><thead><tr><th>Phase</th><th>Description</th></tr></thead><tbody><tr><td><code class="literal">initializing</code></td><td>
The WAL sender process is preparing to begin the backup.
This phase is expected to be very brief.
</td></tr><tr><td><code class="literal">waiting for checkpoint to finish</code></td><td>
The WAL sender process is currently performing
<code class="function">pg_start_backup</code> to prepare to
take a base backup, and waiting for the start-of-backup
checkpoint to finish.
</td></tr><tr><td><code class="literal">estimating backup size</code></td><td>
The WAL sender process is currently estimating the total amount
of database files that will be streamed as a base backup.
</td></tr><tr><td><code class="literal">streaming database files</code></td><td>
The WAL sender process is currently streaming database files
as a base backup.
</td></tr><tr><td><code class="literal">waiting for wal archiving to finish</code></td><td>
The WAL sender process is currently performing
<code class="function">pg_stop_backup</code> to finish the backup,
and waiting for all the WAL files required for the base backup
to be successfully archived.
If either <code class="literal">--wal-method=none</code> or
<code class="literal">--wal-method=stream</code> is specified in
<span class="application">pg_basebackup</span>, the backup will end
when this phase is completed.
</td></tr><tr><td><code class="literal">transferring wal files</code></td><td>
The WAL sender process is currently transferring all WAL logs
generated during the backup. This phase occurs after
<code class="literal">waiting for wal archiving to finish</code> phase if
<code class="literal">--wal-method=fetch</code> is specified in
<span class="application">pg_basebackup</span>. The backup will end
when this phase is completed.
</td></tr></tbody></table></div></div><br class="table-break" /></div><div class="sect2" id="COPY-PROGRESS-REPORTING"><div class="titlepage"><div><div><h3 class="title">28.4.6. COPY Progress Reporting</h3></div></div></div><a id="id-1.6.15.9.8.2" class="indexterm"></a><p>
Whenever <code class="command">COPY</code> is running, the
<code class="structname">pg_stat_progress_copy</code> view will contain one row
for each backend that is currently running a <code class="command">COPY</code> command.
The table below describes the information that will be reported and provides
information about how to interpret it.
</p><div class="table" id="PG-STAT-PROGRESS-COPY-VIEW"><p class="title"><strong>Table 28.46. <code class="structname">pg_stat_progress_copy</code> View</strong></p><div class="table-contents"><table class="table" summary="pg_stat_progress_copy View" border="1"><colgroup><col /></colgroup><thead><tr><th class="catalog_table_entry"><p class="column_definition">
Column Type
</p>
<p>
Description
</p></th></tr></thead><tbody><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">pid</code> <code class="type">integer</code>
</p>
<p>
Process ID of backend.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datid</code> <code class="type">oid</code>
</p>
<p>
OID of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">datname</code> <code class="type">name</code>
</p>
<p>
Name of the database to which this backend is connected.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">relid</code> <code class="type">oid</code>
</p>
<p>
OID of the table on which the <code class="command">COPY</code> command is
executed. It is set to <code class="literal">0</code> if copying from a
<code class="command">SELECT</code> query.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">command</code> <code class="type">text</code>
</p>
<p>
The command that is running: <code class="literal">COPY FROM</code>, or
<code class="literal">COPY TO</code>.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">type</code> <code class="type">text</code>
</p>
<p>
The io type that the data is read from or written to:
<code class="literal">FILE</code>, <code class="literal">PROGRAM</code>,
<code class="literal">PIPE</code> (for <code class="command">COPY FROM STDIN</code> and
<code class="command">COPY TO STDOUT</code>), or <code class="literal">CALLBACK</code>
(used for example during the initial table synchronization in
logical replication).
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">bytes_processed</code> <code class="type">bigint</code>
</p>
<p>
Number of bytes already processed by <code class="command">COPY</code> command.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">bytes_total</code> <code class="type">bigint</code>
</p>
<p>
Size of source file for <code class="command">COPY FROM</code> command in bytes.
It is set to <code class="literal">0</code> if not available.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_processed</code> <code class="type">bigint</code>
</p>
<p>
Number of tuples already processed by <code class="command">COPY</code> command.
</p></td></tr><tr><td class="catalog_table_entry"><p class="column_definition">
<code class="structfield">tuples_excluded</code> <code class="type">bigint</code>
</p>
<p>
Number of tuples not processed because they were excluded by the
<code class="command">WHERE</code> clause of the <code class="command">COPY</code> command.
</p></td></tr></tbody></table></div></div><br class="table-break" /></div></div><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navfooter"><hr></hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="monitoring-locks.html" title="28.3. Viewing Locks">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="monitoring.html" title="Chapter 28. Monitoring Database Activity">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="dynamic-trace.html" title="28.5. Dynamic Tracing">Next</a></td></tr><tr><td width="40%" align="left" valign="top">28.3. Viewing Locks </td><td width="20%" align="center"><a accesskey="h" href="index.html" title="PostgreSQL 15devel Documentation">Home</a></td><td width="40%" align="right" valign="top"> 28.5. Dynamic Tracing</td></tr></table></div></body></html>
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-01-27 21:09 ` Imseih (AWS), Sami <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-01-27 21:09 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
Resending patch as I see the last attachment was not annotated to the commitfest entry.
On 1/26/22, 8:07 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
Attached is the latest patch and associated documentation.
This version addresses the index_ordinal_position column confusion. Rather than displaying the index position, the pg_stat_progress_vacuum view now has 2 new column(s):
index_total - this column will show the total number of indexes to be vacuumed
index_complete_count - this column will show the total number of indexes processed so far. In order to deal with the parallel vacuums, the parallel_workers ( planned workers ) value had to be exposed and each backends performing an index vacuum/cleanup in parallel had to advertise the number of indexes it vacuumed/cleaned. The # of indexes vacuumed for the parallel cleanup can then be derived the pg_stat_progress_vacuum view.
postgres=# \d pg_stat_progress_vacuum
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default
----------------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
relid | oid | | |
phase | text | | |
heap_blks_total | bigint | | |
heap_blks_scanned | bigint | | |
heap_blks_vacuumed | bigint | | |
index_vacuum_count | bigint | | |
max_dead_tuples | bigint | | |
num_dead_tuples | bigint | | |
index_total | bigint | | |. <<<---------------------
index_complete_count | numeric | | |. <<<---------------------
The pg_stat_progress_vacuum_index view includes:
Indexrelid - the currently vacuumed index
Leader_pid - the pid of the leader process. NULL if the process is the leader or vacuum is not parallel
tuples_removed - the amount of indexes tuples removed. The user can use this column to see that the index vacuum has movement.
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
phase | text | | |
leader_pid | bigint | | |
tuples_removed | bigint | | |
On 1/12/22, 9:52 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
On 1/12/22, 1:28 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
> I suppose we could add all of the new columns to
> pg_stat_progress_vacuum and just set columns to NULL as appropriate.
> But is that really better than having a separate view?
To add, since a vacuum can utilize parallel worker processes + the main vacuum process to perform index vacuuming, it made sense to separate the backends doing index vacuum/cleanup in a separate view.
Besides what Nathan suggested, the only other clean option I can think of is to perhaps create a json column in pg_stat_progress_vacuum which will include all the new fields. My concern with this approach is that it will make usability, to flatten the json, difficult for users.
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
> I think the idea was that you can compare it against max_dead_tuples
> and num_dead_tuples to get an estimate of the current cycle progress.
> Otherwise, it just shows that progress is being made.
The main purpose is to really show that the "index vacuum" phase is actually making progress. Note that for certain types of indexes, i.e. GIN/GIST the number of tuples_removed will end up exceeding the number of num_dead_tuples.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
Attachments:
[application/octet-stream] 0004-Expose-progress-for-the-vacuuming-indexes-cleanup-ph.patch (28.5K, ../../[email protected]/2-0004-Expose-progress-for-the-vacuuming-indexes-cleanup-ph.patch)
download | inline diff:
From 33d4116a6b2ea4eef7d00ada279a329854514068 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)"
<[email protected]>
Date: Thu, 27 Jan 2022 01:41:20 +0000
Subject: [PATCH 1/1] Expose progress for the "vacuuming indexes/cleanup" phase
of a VACUUM operation.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/5478DFCD-2333-401A-B2F0-0D186AB09228%40amazon.com
---
doc/src/sgml/monitoring.sgml | 131 ++++++++++++++++++++++++++
src/backend/access/gin/ginvacuum.c | 4 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 11 +++
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 70 ++++++++++----
src/backend/commands/vacuumparallel.c | 56 ++++++++---
src/include/commands/progress.h | 6 ++
src/test/regress/expected/rules.out | 63 ++++++++++---
11 files changed, 309 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 62f2a3332b..b2b33acb66 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -379,6 +379,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_progress_vacuum_index</structname><indexterm><primary>pg_stat_progress_vacuum_index</primary></indexterm></entry>
+ <entry>One row for each backend (including autovacuum worker processes) running
+ the <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase of a
+ <command>VACUUM</command>, showing current progress.
+ See <xref linkend='vacuum-progress-reporting'/>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_cluster</structname><indexterm><primary>pg_stat_progress_cluster</primary></indexterm></entry>
<entry>One row for each backend running
@@ -6269,6 +6278,128 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>index_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The total number of indexes that will be processed by the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>index_complete_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The total number of indexes processed by the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> of the vacuum.
+ For vacuums with mutiple index vacuum cycles, this value will be set to <literal>0</literal>
+ at the start of each cycle.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..73660719b5 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,8 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
+
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index d48c8a4549..84341aa7e7 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -631,6 +631,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1749cc2a47..432229adea 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -340,6 +340,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
vacuum_set_xid_limits(rel,
params->freeze_min_age,
@@ -455,6 +456,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->FreezeLimit = FreezeLimit;
vacrel->MultiXactCutoff = MultiXactCutoff;
+ /* Advertise the total indexes to vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
+
/*
* Call lazy_scan_heap to perform all required heap pruning, index
* vacuuming, and heap vacuuming (plus related processing)
@@ -2010,9 +2014,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_COMPLETED, idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
if (lazy_check_wraparound_failsafe(vacrel))
{
@@ -2419,9 +2427,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_COMPLETED, idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
}
}
else
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 13024af2fa..7455d73d13 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1272,6 +1272,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3cb69b1f87..159605cb3f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1113,22 +1113,60 @@ CREATE VIEW pg_stat_progress_analyze AS
LEFT JOIN pg_database D ON S.datid = D.oid;
CREATE VIEW pg_stat_progress_vacuum AS
- SELECT
- S.pid AS pid, S.datid AS datid, D.datname AS datname,
- S.relid AS relid,
- CASE S.param1 WHEN 0 THEN 'initializing'
- WHEN 1 THEN 'scanning heap'
- WHEN 2 THEN 'vacuuming indexes'
- WHEN 3 THEN 'vacuuming heap'
- WHEN 4 THEN 'cleaning up indexes'
- WHEN 5 THEN 'truncating heap'
- WHEN 6 THEN 'performing final cleanup'
- END AS phase,
- S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
- S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
- FROM pg_stat_get_progress_info('VACUUM') AS S
- LEFT JOIN pg_database D ON S.datid = D.oid;
+ SELECT
+ S.pid,
+ S.datid,
+ S.datname,
+ S.relid,
+ S.phase,
+ S.heap_blks_total,
+ S.heap_blks_scanned,
+ S.heap_blks_vacuumed,
+ S.index_vacuum_count,
+ S.max_dead_tuples,
+ S.num_dead_tuples,
+ S.index_total,
+ S.index_complete_count
+ FROM (SELECT
+ S.pid AS pid, S.datid AS datid, D.datname AS datname,
+ S.relid AS relid, S.param12 as leader_pid,
+ CASE S.param1 WHEN 0 THEN 'initializing'
+ WHEN 1 THEN 'scanning heap'
+ WHEN 2 THEN 'vacuuming indexes'
+ WHEN 3 THEN 'vacuuming heap'
+ WHEN 4 THEN 'cleaning up indexes'
+ WHEN 5 THEN 'truncating heap'
+ WHEN 6 THEN 'performing final cleanup'
+ END AS phase,
+ S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
+ S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS index_total,
+ CASE WHEN S.param1 IN (2,4) AND S.param9 > 0
+ AND S.param10 > 0
+ AND S.param10 > (COUNT(*) OVER (PARTITION BY S.param12) - 1)
+ THEN S.param8 - (COUNT(*) OVER (PARTITION BY S.param12) - 1)
+ ELSE SUM(S.param9) OVER(PARTITION BY S.param12)
+ END AS index_complete_count
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid) AS S
+ WHERE S.pid = S.leader_pid;
+
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid AS pid, S.datid AS datid,
+ D.datname AS datname,
+ S.param11 AS indexrelid,
+ CASE S.param1 WHEN 2 THEN 'vacuuming indexes'
+ WHEN 4 THEN 'cleaning up indexes'
+ END AS phase,
+ CASE WHEN S.param12 = s.pid THEN NULL
+ ELSE S.param12
+ END AS leader_pid,
+ S.param13 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid =D.oid
+ WHERE S.param1 IN (2, 4) AND S.param11 NOT IN (0);
CREATE VIEW pg_stat_progress_cluster AS
SELECT
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..eb4becf900 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,12 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID */
+ int leader_pid;
+
+ /* Number of planned parallel workers */
+ int parallel_workers;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -204,10 +211,10 @@ static int parallel_vacuum_compute_workers(Relation *indrels, int nindexes, int
bool *will_parallel_vacuum);
static void parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scans,
bool vacuum);
-static void parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs);
-static void parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs);
-static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
- PVIndStats *indstats);
+static void parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs, int *idx_vacuumed);
+static void parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs, int *idx_vacuumed);
+static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
+ PVIndStats *indstats, int *idx_vacuumed);
static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_index_scans,
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
@@ -259,6 +266,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
return NULL;
}
+ pgstat_progress_update_param(PROGRESS_VACUUM_WORKERS_PLANNED, parallel_workers);
+
pvs = (ParallelVacuumState *) palloc0(sizeof(ParallelVacuumState));
pvs->indrels = indrels;
pvs->nindexes = nindexes;
@@ -357,6 +366,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->parallel_workers = parallel_workers;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -565,6 +576,7 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
{
int nworkers;
PVIndVacStatus new_status;
+ int idx_vacuumed = 0;
Assert(!IsParallelWorker());
@@ -671,13 +683,13 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
}
/* Vacuum the indexes that can be processed by only leader process */
- parallel_vacuum_process_unsafe_indexes(pvs);
+ parallel_vacuum_process_unsafe_indexes(pvs, &idx_vacuumed);
/*
* Join as a parallel worker. The leader vacuums alone processes all
* parallel-safe indexes in the case where no workers are launched.
*/
- parallel_vacuum_process_safe_indexes(pvs);
+ parallel_vacuum_process_safe_indexes(pvs, &idx_vacuumed);
/*
* Next, accumulate buffer and WAL usage. (This must wait for the workers
@@ -723,7 +735,7 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
* vacuum worker processes to vacuum the indexes in parallel.
*/
static void
-parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs)
+parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs, int *idx_vacuumed)
{
/*
* Increment the active worker count if we are able to launch any worker.
@@ -755,7 +767,8 @@ parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs)
continue;
/* Do vacuum or cleanup of the index */
- parallel_vacuum_process_one_index(pvs, pvs->indrels[idx], indstats);
+ parallel_vacuum_process_one_index(pvs, pvs->indrels[idx], indstats, idx_vacuumed);
+
}
/*
@@ -777,7 +790,7 @@ parallel_vacuum_process_safe_indexes(ParallelVacuumState *pvs)
* enforced by parallel_vacuum_compute_workers().
*/
static void
-parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
+parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs, int *idx_vacuumed)
{
Assert(!IsParallelWorker());
@@ -796,7 +809,8 @@ parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
continue;
/* Do vacuum or cleanup of the index */
- parallel_vacuum_process_one_index(pvs, pvs->indrels[i], indstats);
+ parallel_vacuum_process_one_index(pvs, pvs->indrels[i], indstats, idx_vacuumed);
+
}
/*
@@ -805,6 +819,7 @@ parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
*/
if (VacuumActiveNWorkers)
pg_atomic_sub_fetch_u32(VacuumActiveNWorkers, 1);
+
}
/*
@@ -815,7 +830,7 @@ parallel_vacuum_process_unsafe_indexes(ParallelVacuumState *pvs)
*/
static void
parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
- PVIndStats *indstats)
+ PVIndStats *indstats, int *idx_vacuumed)
{
IndexBulkDeleteResult *istat = NULL;
IndexBulkDeleteResult *istat_res;
@@ -840,13 +855,19 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ (*idx_vacuumed)++;
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ (*idx_vacuumed)++;
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
@@ -885,6 +906,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
pvs->indname = NULL;
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_COMPLETED, *idx_vacuumed);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
}
/*
@@ -943,6 +968,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
int nindexes;
char *sharedquery;
ErrorContextCallback errcallback;
+ int idx_vacuumed = 0;
/*
* A parallel vacuum worker must have only PROC_IN_VACUUM flag since we
@@ -965,6 +991,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM, RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
+ pgstat_progress_update_param(PROGRESS_VACUUM_WORKERS_PLANNED, shared->parallel_workers);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -973,6 +1002,8 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_open_indexes(rel, RowExclusiveLock, &nindexes, &indrels);
Assert(nindexes > 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, nindexes);
+
if (shared->maintenance_work_mem_worker > 0)
maintenance_work_mem = shared->maintenance_work_mem_worker;
@@ -1022,7 +1053,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrStartParallelQuery();
/* Process indexes to perform vacuum/cleanup */
- parallel_vacuum_process_safe_indexes(&pvs);
+ parallel_vacuum_process_safe_indexes(&pvs, &idx_vacuumed);
/* Report buffer/WAL usage during parallel execution */
buffer_usage = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_BUFFER_USAGE, false);
@@ -1036,6 +1067,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
FreeAccessStrategy(pvs.bstrategy);
+ pgstat_progress_end_command();
}
/*
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..83e74e44b3 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,12 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_NUM_INDEX_COMPLETED 8
+#define PROGRESS_VACUUM_WORKERS_PLANNED 9
+#define PROGRESS_VACUUM_INDEXRELID 10
+#define PROGRESS_VACUUM_LEADER_PID 11
+#define PROGRESS_VACUUM_TUPLES_REMOVED 12
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d652f7b5fb..ca2e31cd90 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2007,26 +2007,63 @@ pg_stat_progress_create_index| SELECT s.pid,
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_vacuum| SELECT s.pid,
s.datid,
- d.datname,
+ s.datname,
s.relid,
+ s.phase,
+ s.heap_blks_total,
+ s.heap_blks_scanned,
+ s.heap_blks_vacuumed,
+ s.index_vacuum_count,
+ s.max_dead_tuples,
+ s.num_dead_tuples,
+ s.index_total,
+ s.index_complete_count
+ FROM ( SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.relid,
+ s_1.param12 AS leader_pid,
+ CASE s_1.param1
+ WHEN 0 THEN 'initializing'::text
+ WHEN 1 THEN 'scanning heap'::text
+ WHEN 2 THEN 'vacuuming indexes'::text
+ WHEN 3 THEN 'vacuuming heap'::text
+ WHEN 4 THEN 'cleaning up indexes'::text
+ WHEN 5 THEN 'truncating heap'::text
+ WHEN 6 THEN 'performing final cleanup'::text
+ ELSE NULL::text
+ END AS phase,
+ s_1.param2 AS heap_blks_total,
+ s_1.param3 AS heap_blks_scanned,
+ s_1.param4 AS heap_blks_vacuumed,
+ s_1.param5 AS index_vacuum_count,
+ s_1.param6 AS max_dead_tuples,
+ s_1.param7 AS num_dead_tuples,
+ s_1.param8 AS index_total,
+ CASE
+ WHEN ((s_1.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint])) AND (s_1.param9 > 0) AND (s_1.param10 > 0) AND (s_1.param10 > (count(*) OVER (PARTITION BY s_1.param12) - 1))) THEN ((s_1.param8 - (count(*) OVER (PARTITION BY s_1.param12) - 1)))::numeric
+ ELSE sum(s_1.param9) OVER (PARTITION BY s_1.param12)
+ END AS index_complete_count
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))) s
+ WHERE (s.pid = s.leader_pid);
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.param11 AS indexrelid,
CASE s.param1
- WHEN 0 THEN 'initializing'::text
- WHEN 1 THEN 'scanning heap'::text
WHEN 2 THEN 'vacuuming indexes'::text
- WHEN 3 THEN 'vacuuming heap'::text
WHEN 4 THEN 'cleaning up indexes'::text
- WHEN 5 THEN 'truncating heap'::text
- WHEN 6 THEN 'performing final cleanup'::text
ELSE NULL::text
END AS phase,
- s.param2 AS heap_blks_total,
- s.param3 AS heap_blks_scanned,
- s.param4 AS heap_blks_vacuumed,
- s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ CASE
+ WHEN (s.param12 = s.pid) THEN NULL::bigint
+ ELSE s.param12
+ END AS leader_pid,
+ s.param13 AS tuples_removed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
- LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE ((s.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint])) AND (s.param11 <> 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-02-01 20:33 ` Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-02-01 20:33 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
After speaking with Nathan offline, A few changes have been made to the patch.
As mentioned earlier in the thread, tracking how many indexes are processed in PARALLEL vacuum mode is not very straightforward since only the workers or leader process have ability to inspect the Vacuum shared parallel state.
The latest version of the patch introduces a shared memory to track indexes vacuumed/cleaned by each worker ( or leader ) in a PARALLEL vacuum. In order to present this data in the pg_stat_progress_vacuum view, the value of the new column "indexes_processed" is retrieved from shared memory by pg_stat_get_progress_info. For non-parallel vacuums, the value of "indexes_processed" is retrieved from the backend progress array directly.
The patch also includes the changes to implement the new view pg_stat_progress_vacuum_index which exposes the index being vacuumed/cleaned up.
postgres=# \d+ pg_stat_progress_vacuum ;
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default | Storage | Description
--------------------+---------+-----------+----------+---------+----------+-------------
pid | integer | | | | plain |
datid | oid | | | | plain |
datname | name | | | | plain |
relid | oid | | | | plain |
phase | text | | | | extended |
heap_blks_total | bigint | | | | plain |
heap_blks_scanned | bigint | | | | plain |
heap_blks_vacuumed | bigint | | | | plain |
index_vacuum_count | bigint | | | | plain |
max_dead_tuples | bigint | | | | plain |
num_dead_tuples | bigint | | | | plain |
indexes_total | bigint | | | | plain | <<<-- new column
indexes_processed | bigint | | | | plain | <<<-- new column
<<<--- new view --->>>
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
leader_pid | bigint | | |
phase | text | | |
tuples_removed | bigint | | |
On 1/26/22, 8:07 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
Attached is the latest patch and associated documentation.
This version addresses the index_ordinal_position column confusion. Rather than displaying the index position, the pg_stat_progress_vacuum view now has 2 new column(s):
index_total - this column will show the total number of indexes to be vacuumed
index_complete_count - this column will show the total number of indexes processed so far. In order to deal with the parallel vacuums, the parallel_workers ( planned workers ) value had to be exposed and each backends performing an index vacuum/cleanup in parallel had to advertise the number of indexes it vacuumed/cleaned. The # of indexes vacuumed for the parallel cleanup can then be derived the pg_stat_progress_vacuum view.
postgres=# \d pg_stat_progress_vacuum
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default
----------------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
relid | oid | | |
phase | text | | |
heap_blks_total | bigint | | |
heap_blks_scanned | bigint | | |
heap_blks_vacuumed | bigint | | |
index_vacuum_count | bigint | | |
max_dead_tuples | bigint | | |
num_dead_tuples | bigint | | |
index_total | bigint | | |. <<<---------------------
index_complete_count | numeric | | |. <<<---------------------
The pg_stat_progress_vacuum_index view includes:
Indexrelid - the currently vacuumed index
Leader_pid - the pid of the leader process. NULL if the process is the leader or vacuum is not parallel
tuples_removed - the amount of indexes tuples removed. The user can use this column to see that the index vacuum has movement.
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
phase | text | | |
leader_pid | bigint | | |
tuples_removed | bigint | | |
On 1/12/22, 9:52 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
On 1/12/22, 1:28 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
> I suppose we could add all of the new columns to
> pg_stat_progress_vacuum and just set columns to NULL as appropriate.
> But is that really better than having a separate view?
To add, since a vacuum can utilize parallel worker processes + the main vacuum process to perform index vacuuming, it made sense to separate the backends doing index vacuum/cleanup in a separate view.
Besides what Nathan suggested, the only other clean option I can think of is to perhaps create a json column in pg_stat_progress_vacuum which will include all the new fields. My concern with this approach is that it will make usability, to flatten the json, difficult for users.
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
> I think the idea was that you can compare it against max_dead_tuples
> and num_dead_tuples to get an estimate of the current cycle progress.
> Otherwise, it just shows that progress is being made.
The main purpose is to really show that the "index vacuum" phase is actually making progress. Note that for certain types of indexes, i.e. GIN/GIST the number of tuples_removed will end up exceeding the number of num_dead_tuples.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
Attachments:
[application/octet-stream] 0005-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch (33.0K, ../../[email protected]/2-0005-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch)
download | inline diff:
From f9c0fc69d99a8c681ee3326f12acbf752c66856c Mon Sep 17 00:00:00 2001
From: "Imseih (AWS), Sami" <[email protected]>
Date: Tue, 1 Feb 2022 20:12:18 +0000
Subject: [PATCH 1/1] Expose progress for the "vacuuming indexes" and "cleaning
up indexes" phase of a VACUUM operation.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/5478DFCD-2333-401A-B2F0-0D186AB09228%40amazon.com
---
doc/src/sgml/monitoring.sgml | 121 +++++++++++++
src/backend/access/gin/ginvacuum.c | 3 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 236 ++++++++++++++++++++++++--
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 42 ++++-
src/backend/commands/vacuumparallel.c | 25 +++
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/adt/pgstatfuncs.c | 20 +++
src/include/commands/progress.h | 7 +
src/include/commands/vacuum.h | 8 +
src/include/utils/backend_progress.h | 1 +
src/test/regress/expected/rules.out | 38 ++++-
15 files changed, 499 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 62f2a3332b..0e2532effc 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6269,6 +6269,127 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes processed so far in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of the vacuum.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..1d5d003780 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,7 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..23dacee52e 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -632,6 +632,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f09ad8f271..2d092f194b 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -239,6 +240,25 @@ typedef struct LVSavedErrInfo
VacErrPhase phase;
} LVSavedErrInfo;
+/* Structs for tracking shared Progress information
+ * amongst worker ( and leader ) processes of a vacuum.
+ */
+typedef struct VacOneWorkerProgressInfo
+{
+ int leader_pid;
+ int indexes_processed;
+} VacOneWorkerProgressInfo;
+
+typedef struct VacWorkerProgressInfo
+{
+ int num_vacuums; /* number of active VACUUMS with parallel workers */
+ int max_vacuums; /* max number of VACUUMS with parallel workers */
+ slock_t mutex;
+ VacOneWorkerProgressInfo vacuums[FLEXIBLE_ARRAY_MEMBER];
+} VacWorkerProgressInfo;
+
+static VacWorkerProgressInfo *vacworkerprogress;
+
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params,
@@ -341,6 +361,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
vacuum_set_xid_limits(rel,
params->freeze_min_age,
@@ -404,6 +425,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2000,21 +2025,34 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
return false;
}
- /* Report that we are now vacuuming indexes */
- pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
- PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
-
if (!ParallelVacuumIsActive(vacrel))
{
+ /* Report that we are now vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being vacuumed */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /* For the non-parallel variant of a vacuum, the array position
+ * of the index determines how many indexes are processed so far.
+ * Add 1 to the posititon as this is 0-based array.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+
+ /* Advertise we are done vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2025,9 +2063,28 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
+ /* Report that we are now vacuuming indexes in parallel
+ * and Outsource everything to parallel variant.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL);
+
+ /*
+ * For the parallel variant of a vacuum, we will be populating shared memory
+ * for the index completion progress. This is done with a call to
+ * vacuum_worker_update inside vacuumparallel.c.
+ *
+ * Make sure we are properly cleaning up this shared memory on failure
+ * or we will end up with a leak in the slots.
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2405,33 +2462,55 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
{
Assert(vacrel->nindexes > 0);
- /* Report that we are now cleaning up indexes */
- pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
- PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
-
if (!ParallelVacuumIsActive(vacrel))
{
double reltuples = vacrel->new_rel_tuples;
bool estimated_count =
vacrel->tupcount_pages < vacrel->rel_pages;
+ /* Report that we are now cleaning up indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being cleaned */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /* See the lazy_vacuum_all_indexes comments */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+
+ /* Advertise we are done cleaning indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ /* Report that we are now cleaning up indexes in parallel
+ * and Outsource everything to parallel variant.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL);
+
+ /* See the lazy_vacuum_all_indexes comments */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
vacrel->num_index_scans,
(vacrel->tupcount_pages < vacrel->rel_pages));
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
}
+
}
/*
@@ -3205,3 +3284,136 @@ restore_vacuum_error_info(LVRelState *vacrel,
vacrel->offnum = saved_vacrel->offnum;
vacrel->phase = saved_vacrel->phase;
}
+
+void
+vacuum_worker_end(int leader_pid)
+{
+ SpinLockAcquire(&vacworkerprogress->mutex);
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ VacOneWorkerProgressInfo *vac = &vacworkerprogress->vacuums[i];
+
+ if (vac->leader_pid == leader_pid)
+ {
+ *vac = vacworkerprogress->vacuums[vacworkerprogress->num_vacuums - 1];
+ vacworkerprogress->num_vacuums--;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ break;
+ }
+ }
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
+
+/*
+ * vacuum_worker_end wrapped as an on_shmem_exit callback function
+ */
+void
+vacuum_worker_end_callback(int code, Datum arg)
+{
+ vacuum_worker_end(DatumGetInt32(arg));
+}
+
+/*
+ * vacuum_worker_update sets the number of indexes processed so far
+ * in a parallel vacuum. This routine can be
+ * expanded to other progress tracking amongst parallel
+ * workers ( and leader ).
+ */
+void
+vacuum_worker_update(int leader_pid)
+{
+ VacOneWorkerProgressInfo *vac;
+
+ SpinLockAcquire(&vacworkerprogress->mutex);
+
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ int next_leader_pid;
+
+ vac = &vacworkerprogress->vacuums[i];
+
+ next_leader_pid = vac->leader_pid;
+
+ if (next_leader_pid == leader_pid)
+ {
+ vac->indexes_processed++;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ return;
+ }
+ }
+
+ if (vacworkerprogress->num_vacuums >= vacworkerprogress->max_vacuums)
+ {
+ SpinLockRelease(&vacworkerprogress->mutex);
+ elog(ERROR, "out of vacuum worker progress slots");
+ }
+
+ vac = &vacworkerprogress->vacuums[vacworkerprogress->num_vacuums];
+ vac->leader_pid = leader_pid;
+ vac->indexes_processed = 1;
+ vacworkerprogress->num_vacuums++;
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
+
+/*
+ * vacuum_progress_cb updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+vacuum_progress_cb(Datum *values, int offset)
+{
+ VacOneWorkerProgressInfo *vac;
+ int leader_pid = values[0];
+
+ /* If we are vacuuming in parallel, set the number of indexes vacuumed
+ * from the shared memory counter.
+ * */
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ int next_leader_pid;
+
+ vac = &vacworkerprogress->vacuums[i];
+
+ next_leader_pid = vac->leader_pid;
+
+ if (next_leader_pid == leader_pid)
+ values[PROGRESS_VACUUM_INDEXES_COMPLETED + offset] = vac->indexes_processed;
+ }
+}
+
+/*
+ * VacuumWorkerProgressShmemSize --- report amount of shared memory space needed
+ */
+Size
+VacuumWorkerProgressShmemSize(void)
+{
+ Size size;
+
+ size = offsetof(VacWorkerProgressInfo, vacuums);
+ size = add_size(size, mul_size(MaxBackends, sizeof(VacOneWorkerProgressInfo)));
+ return size;
+}
+
+/*
+ * VacuumWorkerProgressShmemInit --- initialize this module's shared memory
+ */
+void
+VacuumWorkerProgressShmemInit(void)
+{
+ bool found;
+
+ vacworkerprogress = (VacWorkerProgressInfo *) ShmemInitStruct("Vacuum Worker Progress Stats",
+ VacuumWorkerProgressShmemSize(),
+ &found);
+
+ if (!IsUnderPostmaster)
+ {
+ /* Initialize shared memory area */
+ Assert(!found);
+
+ vacworkerprogress->max_vacuums = MaxBackends;
+ SpinLockInit(&vacworkerprogress->mutex);
+ }
+ else
+ Assert(found);
+}
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c9b4964c1e..09edf49082 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1273,6 +1273,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3cb69b1f87..0f91f66c73 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1123,13 +1123,53 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 4 THEN 'cleaning up indexes'
WHEN 5 THEN 'truncating heap'
WHEN 6 THEN 'performing final cleanup'
+ WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid,
+ S.datid,
+ S.datname,
+ S.indexrelid,
+ S.leader_pid,
+ CASE S.phase WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
+ END AS phase,
+ S.tuples_removed
+ FROM (
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 IN (2, 4, 7, 8) AND S.param10 > 0
+ UNION ALL
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM_PARALLEL') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ ) AS S
+ WHERE S.phase IN (2, 4, 7, 8) AND S.indexrelid > 0;
+
CREATE VIEW pg_stat_progress_cluster AS
SELECT
S.pid AS pid,
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..1332203047 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -716,6 +721,7 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
VacuumSharedCostBalance = NULL;
VacuumActiveNWorkers = NULL;
}
+
}
/*
@@ -840,13 +846,24 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ /* Advertise the index we are cleaning or vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ /* Report that we are now vacuuming indexes in parallel */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ /* Report that we are now cleaning indexes in parallel */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
@@ -881,6 +898,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
*/
indstats->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+ /* Advertise we are no longer vacuuming/cleaning an index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
@@ -965,6 +986,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL,
+ RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -1036,6 +1060,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
FreeAccessStrategy(pvs.bstrategy);
+ pgstat_progress_end_command();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 9f26e41c46..8682578ac3 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -23,6 +23,7 @@
#include "access/syncscan.h"
#include "access/twophase.h"
#include "commands/async.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -143,6 +144,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, VacuumWorkerProgressShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -293,6 +295,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ VacuumWorkerProgressShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 15cb17ace4..5dece71596 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,7 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/vacuum.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -452,6 +453,14 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
/*
* Returns command progress information for the named command.
+ *
+ * A command type can optionally define a callback function
+ * which will derive Datum values rather than use values
+ * directly from the backends progress array.
+ *
+ * Derived values are useful to calculate values form multiple backends
+ * as is the case with parallel operations, in which progress values
+ * are calculated form multiple workers.
*/
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
@@ -466,6 +475,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
MemoryContext per_query_ctx;
MemoryContext oldcontext;
+ void (*callback)(Datum *, int) = NULL;
/* check to see if caller supports us returning a tuplestore */
if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -483,7 +493,14 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ {
cmdtype = PROGRESS_COMMAND_VACUUM;
+ callback = vacuum_progress_cb;
+ }
+ else if (pg_strcasecmp(cmd, "VACUUM_PARALLEL") == 0)
+ {
+ cmdtype = PROGRESS_COMMAND_VACUUM_PARALLEL;
+ }
else if (pg_strcasecmp(cmd, "ANALYZE") == 0)
cmdtype = PROGRESS_COMMAND_ANALYZE;
else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
@@ -552,6 +569,9 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
nulls[i + 3] = true;
}
+ if (callback)
+ callback(values, 3);
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..37659882ee 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,11 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
+#define PROGRESS_VACUUM_INDEXRELID 9
+#define PROGRESS_VACUUM_TUPLES_REMOVED 10
+#define PROGRESS_VACUUM_LEADER_PID 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
@@ -33,6 +38,8 @@
#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL 7
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL 8
/* Progress parameters for analyze */
#define PROGRESS_ANALYZE_PHASE 0
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5d0bdfa427..119924694d 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -336,4 +336,12 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+/* in commands/vacuumparallel.c */
+extern Size VacuumWorkerProgressShmemSize(void);
+extern void VacuumWorkerProgressShmemInit(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_progress_cb(Datum *values, int offset);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+
#endif /* VACUUM_H */
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..4651e45c40 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -23,6 +23,7 @@ typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_VACUUM_PARALLEL,
PROGRESS_COMMAND_ANALYZE,
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d652f7b5fb..0fafe60e53 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2017,6 +2017,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
WHEN 4 THEN 'cleaning up indexes'::text
WHEN 5 THEN 'truncating heap'::text
WHEN 6 THEN 'performing final cleanup'::text
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
ELSE NULL::text
END AS phase,
s.param2 AS heap_blks_total,
@@ -2024,9 +2026,43 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ s.datname,
+ s.indexrelid,
+ s.leader_pid,
+ CASE s.phase
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
+ ELSE NULL::text
+ END AS phase,
+ s.tuples_removed
+ FROM ( SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))
+ WHERE ((s_1.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s_1.param10 > 0))
+ UNION ALL
+ SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM_PARALLEL'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))) s
+ WHERE ((s.phase = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s.indexrelid > 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-02-10 19:39 ` Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-02-10 19:39 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
Attached is the latest version of the patch to deal with the changes in the recent commit aa64f23b02924724eafbd9eadbf26d85df30a12b
On 2/1/22, 2:32 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
After speaking with Nathan offline, A few changes have been made to the patch.
As mentioned earlier in the thread, tracking how many indexes are processed in PARALLEL vacuum mode is not very straightforward since only the workers or leader process have ability to inspect the Vacuum shared parallel state.
The latest version of the patch introduces a shared memory to track indexes vacuumed/cleaned by each worker ( or leader ) in a PARALLEL vacuum. In order to present this data in the pg_stat_progress_vacuum view, the value of the new column "indexes_processed" is retrieved from shared memory by pg_stat_get_progress_info. For non-parallel vacuums, the value of "indexes_processed" is retrieved from the backend progress array directly.
The patch also includes the changes to implement the new view pg_stat_progress_vacuum_index which exposes the index being vacuumed/cleaned up.
postgres=# \d+ pg_stat_progress_vacuum ;
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default | Storage | Description
--------------------+---------+-----------+----------+---------+----------+-------------
pid | integer | | | | plain |
datid | oid | | | | plain |
datname | name | | | | plain |
relid | oid | | | | plain |
phase | text | | | | extended |
heap_blks_total | bigint | | | | plain |
heap_blks_scanned | bigint | | | | plain |
heap_blks_vacuumed | bigint | | | | plain |
index_vacuum_count | bigint | | | | plain |
max_dead_tuples | bigint | | | | plain |
num_dead_tuples | bigint | | | | plain |
indexes_total | bigint | | | | plain | <<<-- new column
indexes_processed | bigint | | | | plain | <<<-- new column
<<<--- new view --->>>
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
leader_pid | bigint | | |
phase | text | | |
tuples_removed | bigint | | |
On 1/26/22, 8:07 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
Attached is the latest patch and associated documentation.
This version addresses the index_ordinal_position column confusion. Rather than displaying the index position, the pg_stat_progress_vacuum view now has 2 new column(s):
index_total - this column will show the total number of indexes to be vacuumed
index_complete_count - this column will show the total number of indexes processed so far. In order to deal with the parallel vacuums, the parallel_workers ( planned workers ) value had to be exposed and each backends performing an index vacuum/cleanup in parallel had to advertise the number of indexes it vacuumed/cleaned. The # of indexes vacuumed for the parallel cleanup can then be derived the pg_stat_progress_vacuum view.
postgres=# \d pg_stat_progress_vacuum
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default
----------------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
relid | oid | | |
phase | text | | |
heap_blks_total | bigint | | |
heap_blks_scanned | bigint | | |
heap_blks_vacuumed | bigint | | |
index_vacuum_count | bigint | | |
max_dead_tuples | bigint | | |
num_dead_tuples | bigint | | |
index_total | bigint | | |. <<<---------------------
index_complete_count | numeric | | |. <<<---------------------
The pg_stat_progress_vacuum_index view includes:
Indexrelid - the currently vacuumed index
Leader_pid - the pid of the leader process. NULL if the process is the leader or vacuum is not parallel
tuples_removed - the amount of indexes tuples removed. The user can use this column to see that the index vacuum has movement.
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
phase | text | | |
leader_pid | bigint | | |
tuples_removed | bigint | | |
On 1/12/22, 9:52 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
On 1/12/22, 1:28 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
> I suppose we could add all of the new columns to
> pg_stat_progress_vacuum and just set columns to NULL as appropriate.
> But is that really better than having a separate view?
To add, since a vacuum can utilize parallel worker processes + the main vacuum process to perform index vacuuming, it made sense to separate the backends doing index vacuum/cleanup in a separate view.
Besides what Nathan suggested, the only other clean option I can think of is to perhaps create a json column in pg_stat_progress_vacuum which will include all the new fields. My concern with this approach is that it will make usability, to flatten the json, difficult for users.
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
> I think the idea was that you can compare it against max_dead_tuples
> and num_dead_tuples to get an estimate of the current cycle progress.
> Otherwise, it just shows that progress is being made.
The main purpose is to really show that the "index vacuum" phase is actually making progress. Note that for certain types of indexes, i.e. GIN/GIST the number of tuples_removed will end up exceeding the number of num_dead_tuples.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
Attachments:
[application/octet-stream] 0006-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch (32.9K, ../../[email protected]/2-0006-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch)
download | inline diff:
From 4460de90eda020c5e49f0ba264282a2a4bd6565d Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 10 Feb 2022 19:28:27 +0000
Subject: [PATCH 1/1] Expose progress for the "vacuuming indexes" and "cleaning
up indexes" phase of a VACUUM operation.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
---
doc/src/sgml/monitoring.sgml | 121 +++++++++++++
src/backend/access/gin/ginvacuum.c | 3 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 236 ++++++++++++++++++++++++--
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 42 ++++-
src/backend/commands/vacuumparallel.c | 25 +++
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/adt/pgstatfuncs.c | 20 +++
src/include/commands/progress.h | 7 +
src/include/commands/vacuum.h | 8 +
src/include/utils/backend_progress.h | 1 +
src/test/regress/expected/rules.out | 38 ++++-
15 files changed, 499 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 62f2a3332b..0e2532effc 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6269,6 +6269,127 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes processed so far in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of the vacuum.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..1d5d003780 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,7 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..23dacee52e 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -632,6 +632,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f09ad8f271..fb8c9b1030 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -239,6 +240,25 @@ typedef struct LVSavedErrInfo
VacErrPhase phase;
} LVSavedErrInfo;
+/* Structs for tracking shared Progress information
+ * amongst worker ( and leader ) processes of a vacuum.
+ */
+typedef struct VacOneWorkerProgressInfo
+{
+ int leader_pid;
+ int indexes_processed;
+} VacOneWorkerProgressInfo;
+
+typedef struct VacWorkerProgressInfo
+{
+ int num_vacuums; /* number of active VACUUMS with parallel workers */
+ int max_vacuums; /* max number of VACUUMS with parallel workers */
+ slock_t mutex;
+ VacOneWorkerProgressInfo vacuums[FLEXIBLE_ARRAY_MEMBER];
+} VacWorkerProgressInfo;
+
+static VacWorkerProgressInfo *vacworkerprogress;
+
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params,
@@ -341,6 +361,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
vacuum_set_xid_limits(rel,
params->freeze_min_age,
@@ -404,6 +425,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2000,21 +2025,34 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
return false;
}
- /* Report that we are now vacuuming indexes */
- pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
- PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
-
if (!ParallelVacuumIsActive(vacrel))
{
+ /* Report that we are now vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being vacuumed */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /* For the non-parallel variant of a vacuum, the array position
+ * of the index determines how many indexes are processed so far.
+ * Add 1 to the posititon as this is 0-based array.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+
+ /* Advertise we are done vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2025,9 +2063,28 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
+ /* Report that we are now vacuuming indexes in parallel
+ * and Outsource everything to parallel variant.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL);
+
+ /*
+ * For the parallel variant of a vacuum, we will be populating shared memory
+ * for the index completion progress. This is done with a call to
+ * vacuum_worker_update inside vacuumparallel.c.
+ *
+ * Make sure we are properly cleaning up this shared memory on failure
+ * or we will end up with a leak in the slots.
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2405,33 +2462,55 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
{
Assert(vacrel->nindexes > 0);
- /* Report that we are now cleaning up indexes */
- pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
- PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
-
if (!ParallelVacuumIsActive(vacrel))
{
double reltuples = vacrel->new_rel_tuples;
bool estimated_count =
vacrel->tupcount_pages < vacrel->rel_pages;
+ /* Report that we are now cleaning up indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being cleaned */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /* See the lazy_vacuum_all_indexes comments */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+
+ /* Advertise we are done cleaning indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ /* Report that we are now cleaning up indexes in parallel
+ * and Outsource everything to parallel variant.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL);
+
+ /* See the lazy_vacuum_all_indexes comments */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
vacrel->num_index_scans,
(vacrel->tupcount_pages < vacrel->rel_pages));
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
}
+
}
/*
@@ -3205,3 +3284,136 @@ restore_vacuum_error_info(LVRelState *vacrel,
vacrel->offnum = saved_vacrel->offnum;
vacrel->phase = saved_vacrel->phase;
}
+
+void
+vacuum_worker_end(int leader_pid)
+{
+ SpinLockAcquire(&vacworkerprogress->mutex);
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ VacOneWorkerProgressInfo *vac = &vacworkerprogress->vacuums[i];
+
+ if (vac->leader_pid == leader_pid)
+ {
+ *vac = vacworkerprogress->vacuums[vacworkerprogress->num_vacuums - 1];
+ vacworkerprogress->num_vacuums--;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ break;
+ }
+ }
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
+
+/*
+ * vacuum_worker_end wrapped as an on_shmem_exit callback function
+ */
+void
+vacuum_worker_end_callback(int code, Datum arg)
+{
+ vacuum_worker_end(DatumGetInt32(arg));
+}
+
+/*
+ * vacuum_worker_update sets the number of indexes processed so far
+ * in a parallel vacuum. This routine can be
+ * expanded to other progress tracking amongst parallel
+ * workers ( and leader ).
+ */
+void
+vacuum_worker_update(int leader_pid)
+{
+ VacOneWorkerProgressInfo *vac;
+
+ SpinLockAcquire(&vacworkerprogress->mutex);
+
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ int next_leader_pid;
+
+ vac = &vacworkerprogress->vacuums[i];
+
+ next_leader_pid = vac->leader_pid;
+
+ if (next_leader_pid == leader_pid)
+ {
+ vac->indexes_processed++;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ return;
+ }
+ }
+
+ if (vacworkerprogress->num_vacuums >= vacworkerprogress->max_vacuums)
+ {
+ SpinLockRelease(&vacworkerprogress->mutex);
+ elog(ERROR, "out of vacuum worker progress slots");
+ }
+
+ vac = &vacworkerprogress->vacuums[vacworkerprogress->num_vacuums];
+ vac->leader_pid = leader_pid;
+ vac->indexes_processed = 1;
+ vacworkerprogress->num_vacuums++;
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
+
+/*
+ * vacuum_progress_cb updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+vacuum_progress_cb(Datum *values, int offset)
+{
+ VacOneWorkerProgressInfo *vac;
+ int leader_pid = values[0];
+
+ /* If we are vacuuming in parallel, set the number of indexes vacuumed
+ * from the shared memory counter.
+ * */
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ int next_leader_pid;
+
+ vac = &vacworkerprogress->vacuums[i];
+
+ next_leader_pid = vac->leader_pid;
+
+ if (next_leader_pid == leader_pid)
+ values[PROGRESS_VACUUM_INDEXES_COMPLETED + offset] = vac->indexes_processed;
+ }
+}
+
+/*
+ * VacuumWorkerProgressShmemSize --- report amount of shared memory space needed
+ */
+Size
+VacuumWorkerProgressShmemSize(void)
+{
+ Size size;
+
+ size = offsetof(VacWorkerProgressInfo, vacuums);
+ size = add_size(size, mul_size(GetMaxBackends(), sizeof(VacOneWorkerProgressInfo)));
+ return size;
+}
+
+/*
+ * VacuumWorkerProgressShmemInit --- initialize this module's shared memory
+ */
+void
+VacuumWorkerProgressShmemInit(void)
+{
+ bool found;
+
+ vacworkerprogress = (VacWorkerProgressInfo *) ShmemInitStruct("Vacuum Worker Progress Stats",
+ VacuumWorkerProgressShmemSize(),
+ &found);
+
+ if (!IsUnderPostmaster)
+ {
+ /* Initialize shared memory area */
+ Assert(!found);
+
+ vacworkerprogress->max_vacuums = GetMaxBackends();
+ SpinLockInit(&vacworkerprogress->mutex);
+ }
+ else
+ Assert(found);
+}
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c9b4964c1e..09edf49082 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1273,6 +1273,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3cb69b1f87..0f91f66c73 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1123,13 +1123,53 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 4 THEN 'cleaning up indexes'
WHEN 5 THEN 'truncating heap'
WHEN 6 THEN 'performing final cleanup'
+ WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid,
+ S.datid,
+ S.datname,
+ S.indexrelid,
+ S.leader_pid,
+ CASE S.phase WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
+ END AS phase,
+ S.tuples_removed
+ FROM (
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 IN (2, 4, 7, 8) AND S.param10 > 0
+ UNION ALL
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM_PARALLEL') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ ) AS S
+ WHERE S.phase IN (2, 4, 7, 8) AND S.indexrelid > 0;
+
CREATE VIEW pg_stat_progress_cluster AS
SELECT
S.pid AS pid,
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..1332203047 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -716,6 +721,7 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
VacuumSharedCostBalance = NULL;
VacuumActiveNWorkers = NULL;
}
+
}
/*
@@ -840,13 +846,24 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ /* Advertise the index we are cleaning or vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ /* Report that we are now vacuuming indexes in parallel */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ /* Report that we are now cleaning indexes in parallel */
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
@@ -881,6 +898,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
*/
indstats->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+ /* Advertise we are no longer vacuuming/cleaning an index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
@@ -965,6 +986,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL,
+ RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -1036,6 +1060,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
FreeAccessStrategy(pvs.bstrategy);
+ pgstat_progress_end_command();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 9f26e41c46..8682578ac3 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -23,6 +23,7 @@
#include "access/syncscan.h"
#include "access/twophase.h"
#include "commands/async.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -143,6 +144,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, VacuumWorkerProgressShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -293,6 +295,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ VacuumWorkerProgressShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 15cb17ace4..5dece71596 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,7 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/vacuum.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -452,6 +453,14 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
/*
* Returns command progress information for the named command.
+ *
+ * A command type can optionally define a callback function
+ * which will derive Datum values rather than use values
+ * directly from the backends progress array.
+ *
+ * Derived values are useful to calculate values form multiple backends
+ * as is the case with parallel operations, in which progress values
+ * are calculated form multiple workers.
*/
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
@@ -466,6 +475,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
MemoryContext per_query_ctx;
MemoryContext oldcontext;
+ void (*callback)(Datum *, int) = NULL;
/* check to see if caller supports us returning a tuplestore */
if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -483,7 +493,14 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ {
cmdtype = PROGRESS_COMMAND_VACUUM;
+ callback = vacuum_progress_cb;
+ }
+ else if (pg_strcasecmp(cmd, "VACUUM_PARALLEL") == 0)
+ {
+ cmdtype = PROGRESS_COMMAND_VACUUM_PARALLEL;
+ }
else if (pg_strcasecmp(cmd, "ANALYZE") == 0)
cmdtype = PROGRESS_COMMAND_ANALYZE;
else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
@@ -552,6 +569,9 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
nulls[i + 3] = true;
}
+ if (callback)
+ callback(values, 3);
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..37659882ee 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,11 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
+#define PROGRESS_VACUUM_INDEXRELID 9
+#define PROGRESS_VACUUM_TUPLES_REMOVED 10
+#define PROGRESS_VACUUM_LEADER_PID 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
@@ -33,6 +38,8 @@
#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL 7
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL 8
/* Progress parameters for analyze */
#define PROGRESS_ANALYZE_PHASE 0
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5d0bdfa427..119924694d 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -336,4 +336,12 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+/* in commands/vacuumparallel.c */
+extern Size VacuumWorkerProgressShmemSize(void);
+extern void VacuumWorkerProgressShmemInit(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_progress_cb(Datum *values, int offset);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+
#endif /* VACUUM_H */
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..4651e45c40 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -23,6 +23,7 @@ typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_VACUUM_PARALLEL,
PROGRESS_COMMAND_ANALYZE,
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 1420288d67..35193dc6b7 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1995,6 +1995,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
WHEN 4 THEN 'cleaning up indexes'::text
WHEN 5 THEN 'truncating heap'::text
WHEN 6 THEN 'performing final cleanup'::text
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
ELSE NULL::text
END AS phase,
s.param2 AS heap_blks_total,
@@ -2002,9 +2004,43 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ s.datname,
+ s.indexrelid,
+ s.leader_pid,
+ CASE s.phase
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
+ ELSE NULL::text
+ END AS phase,
+ s.tuples_removed
+ FROM ( SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))
+ WHERE ((s_1.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s_1.param10 > 0))
+ UNION ALL
+ SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM_PARALLEL'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))) s
+ WHERE ((s.phase = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s.indexrelid > 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-02-17 13:52 ` Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-02-17 13:52 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
The change has been broken up as 3 separate patches.
0007-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch - Introduces 2 new columns to pg_stat_progress_vacuum, indexes_total and indexes_processed. These 2 columns will provide progress on the index vacuuming/cleanup.
0001-Expose-the-index-being-processed-in-the-vacuuming-in.patch - Introduces a new view called pg_stat_prgoress_vacuum_index. This view tracks the index being vacuumed/cleaned and the total number of index tuples removed.
0001-Rename-index_vacuum_count-to-index_vacuum_cycle_coun.patch - Renames the existing index_vacuum_count to index_vacuum_cycle_count in pg_stat_progress_vacuum. Due to the other changes, it makes sense to include "cycle" in the column name to be crystal clear that the column refers to the index cycle count.
Thanks
On 2/10/22, 1:39 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
Attached is the latest version of the patch to deal with the changes in the recent commit aa64f23b02924724eafbd9eadbf26d85df30a12b
On 2/1/22, 2:32 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
After speaking with Nathan offline, A few changes have been made to the patch.
As mentioned earlier in the thread, tracking how many indexes are processed in PARALLEL vacuum mode is not very straightforward since only the workers or leader process have ability to inspect the Vacuum shared parallel state.
The latest version of the patch introduces a shared memory to track indexes vacuumed/cleaned by each worker ( or leader ) in a PARALLEL vacuum. In order to present this data in the pg_stat_progress_vacuum view, the value of the new column "indexes_processed" is retrieved from shared memory by pg_stat_get_progress_info. For non-parallel vacuums, the value of "indexes_processed" is retrieved from the backend progress array directly.
The patch also includes the changes to implement the new view pg_stat_progress_vacuum_index which exposes the index being vacuumed/cleaned up.
postgres=# \d+ pg_stat_progress_vacuum ;
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default | Storage | Description
--------------------+---------+-----------+----------+---------+----------+-------------
pid | integer | | | | plain |
datid | oid | | | | plain |
datname | name | | | | plain |
relid | oid | | | | plain |
phase | text | | | | extended |
heap_blks_total | bigint | | | | plain |
heap_blks_scanned | bigint | | | | plain |
heap_blks_vacuumed | bigint | | | | plain |
index_vacuum_count | bigint | | | | plain |
max_dead_tuples | bigint | | | | plain |
num_dead_tuples | bigint | | | | plain |
indexes_total | bigint | | | | plain | <<<-- new column
indexes_processed | bigint | | | | plain | <<<-- new column
<<<--- new view --->>>
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
leader_pid | bigint | | |
phase | text | | |
tuples_removed | bigint | | |
On 1/26/22, 8:07 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
Attached is the latest patch and associated documentation.
This version addresses the index_ordinal_position column confusion. Rather than displaying the index position, the pg_stat_progress_vacuum view now has 2 new column(s):
index_total - this column will show the total number of indexes to be vacuumed
index_complete_count - this column will show the total number of indexes processed so far. In order to deal with the parallel vacuums, the parallel_workers ( planned workers ) value had to be exposed and each backends performing an index vacuum/cleanup in parallel had to advertise the number of indexes it vacuumed/cleaned. The # of indexes vacuumed for the parallel cleanup can then be derived the pg_stat_progress_vacuum view.
postgres=# \d pg_stat_progress_vacuum
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default
----------------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
relid | oid | | |
phase | text | | |
heap_blks_total | bigint | | |
heap_blks_scanned | bigint | | |
heap_blks_vacuumed | bigint | | |
index_vacuum_count | bigint | | |
max_dead_tuples | bigint | | |
num_dead_tuples | bigint | | |
index_total | bigint | | |. <<<---------------------
index_complete_count | numeric | | |. <<<---------------------
The pg_stat_progress_vacuum_index view includes:
Indexrelid - the currently vacuumed index
Leader_pid - the pid of the leader process. NULL if the process is the leader or vacuum is not parallel
tuples_removed - the amount of indexes tuples removed. The user can use this column to see that the index vacuum has movement.
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
phase | text | | |
leader_pid | bigint | | |
tuples_removed | bigint | | |
On 1/12/22, 9:52 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
On 1/12/22, 1:28 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
> I suppose we could add all of the new columns to
> pg_stat_progress_vacuum and just set columns to NULL as appropriate.
> But is that really better than having a separate view?
To add, since a vacuum can utilize parallel worker processes + the main vacuum process to perform index vacuuming, it made sense to separate the backends doing index vacuum/cleanup in a separate view.
Besides what Nathan suggested, the only other clean option I can think of is to perhaps create a json column in pg_stat_progress_vacuum which will include all the new fields. My concern with this approach is that it will make usability, to flatten the json, difficult for users.
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
> I think the idea was that you can compare it against max_dead_tuples
> and num_dead_tuples to get an estimate of the current cycle progress.
> Otherwise, it just shows that progress is being made.
The main purpose is to really show that the "index vacuum" phase is actually making progress. Note that for certain types of indexes, i.e. GIN/GIST the number of tuples_removed will end up exceeding the number of num_dead_tuples.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
Attachments:
[application/octet-stream] 0007-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch (17.2K, ../../[email protected]/2-0007-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch)
download | inline diff:
From 76d8a1bfd5207d28a4e9fe98a0e1ea7c096d70aa Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 17 Feb 2022 04:21:04 +0000
Subject: [PATCH 1/1] Expose progress for the "vacuuming indexes" and "cleaning
up indexes" phase of a VACUUM operation.
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
---
doc/src/sgml/monitoring.sgml | 22 +++
src/backend/access/heap/vacuumlazy.c | 207 +++++++++++++++++++++++++-
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 7 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/adt/pgstatfuncs.c | 12 ++
src/include/commands/progress.h | 2 +
src/include/commands/vacuum.h | 8 +
src/test/regress/expected/rules.out | 4 +-
9 files changed, 261 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index bf7625d988..04440dfa88 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6278,6 +6278,28 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase.
+ At the start of an index vacuum cycle, this value is set to <literal>0</literal>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 242511a235..1198677bc0 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -245,6 +246,26 @@ typedef struct LVSavedErrInfo
VacErrPhase phase;
} LVSavedErrInfo;
+/*
+ * Structs for tracking shared Progress information
+ * amongst worker ( and leader ) processes of a vacuum.
+ */
+typedef struct VacOneWorkerProgressInfo
+{
+ int leader_pid;
+ int indexes_processed;
+} VacOneWorkerProgressInfo;
+
+typedef struct VacWorkerProgressInfo
+{
+ int num_vacuums; /* number of active VACUUMS with parallel workers */
+ int max_vacuums; /* max number of VACUUMS with parallel workers */
+ slock_t mutex;
+ VacOneWorkerProgressInfo vacuums[FLEXIBLE_ARRAY_MEMBER];
+} VacWorkerProgressInfo;
+
+static VacWorkerProgressInfo *vacworkerprogress;
+
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
@@ -420,6 +441,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2328,6 +2351,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /*
+ * For the non-parallel variant of a vacuum, the array position
+ * of the index determines how many indexes are processed so far.
+ * Add 1 to the posititon as vacrel->nindexes is a 0-based array.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2338,9 +2368,20 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
- vacrel->num_index_scans);
+ /*
+ * parallel_vacuum_bulkdel_all_indexes will call vacuum_worker_update
+ * which updates shared memory for the index progress. To ensure shared
+ * memory cleanup, do the work with PG_ENSURE_ERROR_CLEANUP.
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs,
+ vacrel->old_live_tuples,
+ vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2350,6 +2391,12 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
allindexes = false;
}
+ /*
+ * We're done with index vacuuming.
+ * Set the total number of indexes completed as the total number of indexes
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, vacrel->nindexes) ;
+
/*
* We delete all LP_DEAD items from the first heap pass in all indexes on
* each call here (except calls where we choose to do the failsafe). This
@@ -2675,15 +2722,34 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /* See the lazy_vacuum_all_indexes comments */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ /*
+ * Outsource everything to parallel variant
+ *
+ * See the lazy_vacuum_all_indexes comments
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
vacrel->num_index_scans,
(vacrel->scanned_pages < vacrel->rel_pages));
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
}
+
+ /*
+ * We're done with index cleanup.
+ * Set the total number of indexes completed as the total number of indexes
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, vacrel->nindexes) ;
+
}
/*
@@ -3464,3 +3530,134 @@ restore_vacuum_error_info(LVRelState *vacrel,
vacrel->offnum = saved_vacrel->offnum;
vacrel->phase = saved_vacrel->phase;
}
+
+void
+vacuum_worker_end(int leader_pid)
+{
+ SpinLockAcquire(&vacworkerprogress->mutex);
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ VacOneWorkerProgressInfo *vac = &vacworkerprogress->vacuums[i];
+
+ if (vac->leader_pid == leader_pid)
+ {
+ *vac = vacworkerprogress->vacuums[vacworkerprogress->num_vacuums - 1];
+ vacworkerprogress->num_vacuums--;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ break;
+ }
+ }
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
+
+/*
+ * vacuum_worker_end wrapped as an on_shmem_exit callback function
+ */
+void
+vacuum_worker_end_callback(int code, Datum arg)
+{
+ vacuum_worker_end(DatumGetInt32(arg));
+}
+
+/*
+ * vacuum_worker_update sets the number of indexes processed so far
+ * in a parallel vacuum.
+ */
+void
+vacuum_worker_update(int leader_pid)
+{
+ VacOneWorkerProgressInfo *vac;
+
+ SpinLockAcquire(&vacworkerprogress->mutex);
+
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ int next_leader_pid;
+
+ vac = &vacworkerprogress->vacuums[i];
+
+ next_leader_pid = vac->leader_pid;
+
+ if (next_leader_pid == leader_pid)
+ {
+ vac->indexes_processed++;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ return;
+ }
+ }
+
+ if (vacworkerprogress->num_vacuums >= vacworkerprogress->max_vacuums)
+ {
+ SpinLockRelease(&vacworkerprogress->mutex);
+ elog(ERROR, "out of vacuum worker progress slots");
+ }
+
+ vac = &vacworkerprogress->vacuums[vacworkerprogress->num_vacuums];
+ vac->leader_pid = leader_pid;
+ vac->indexes_processed = 1;
+ vacworkerprogress->num_vacuums++;
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
+
+/*
+ * vacuum_progress_cb updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+vacuum_progress_cb(Datum *values, int offset)
+{
+ VacOneWorkerProgressInfo *vac;
+ int leader_pid = values[0];
+
+ /* If we are vacuuming in parallel, set the number of indexes vacuumed
+ * from the shared memory counter.
+ * */
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ int next_leader_pid;
+
+ vac = &vacworkerprogress->vacuums[i];
+
+ next_leader_pid = vac->leader_pid;
+
+ if (next_leader_pid == leader_pid)
+ values[PROGRESS_VACUUM_INDEXES_COMPLETED + offset] = vac->indexes_processed;
+ }
+}
+
+/*
+ * VacuumWorkerProgressShmemSize --- report amount of shared memory space needed
+ */
+Size
+VacuumWorkerProgressShmemSize(void)
+{
+ Size size;
+
+ size = offsetof(VacWorkerProgressInfo, vacuums);
+ size = add_size(size, mul_size(GetMaxBackends(), sizeof(VacOneWorkerProgressInfo)));
+ return size;
+}
+
+/*
+ * VacuumWorkerProgressShmemInit --- initialize this module's shared memory
+ */
+void
+VacuumWorkerProgressShmemInit(void)
+{
+ bool found;
+
+ vacworkerprogress = (VacWorkerProgressInfo *) ShmemInitStruct("Vacuum Worker Progress Stats",
+ VacuumWorkerProgressShmemSize(),
+ &found);
+
+ if (!IsUnderPostmaster)
+ {
+ /* Initialize shared memory area */
+ Assert(!found);
+
+ vacworkerprogress->max_vacuums = GetMaxBackends();
+ SpinLockInit(&vacworkerprogress->mutex);
+ }
+ else
+ Assert(found);
+}
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3cb69b1f87..eaa0508c0b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1126,7 +1126,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..9b465e12cc 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -844,9 +849,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index cd4ebe2fc5..a4bd6a14a3 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -24,6 +24,7 @@
#include "access/twophase.h"
#include "access/xlogrecovery.h"
#include "commands/async.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -145,6 +146,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, VacuumWorkerProgressShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -296,6 +298,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ VacuumWorkerProgressShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 15cb17ace4..3d90b23ce4 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,7 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/vacuum.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -452,6 +453,10 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
/*
* Returns command progress information for the named command.
+ *
+ * A command type can optionally define a callback function
+ * which will derive Datum values rather than use values
+ * directly from the backends progress array.
*/
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
@@ -466,6 +471,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
MemoryContext per_query_ctx;
MemoryContext oldcontext;
+ void (*callback)(Datum *, int) = NULL;
/* check to see if caller supports us returning a tuplestore */
if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -483,7 +489,10 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ {
cmdtype = PROGRESS_COMMAND_VACUUM;
+ callback = vacuum_progress_cb;
+ }
else if (pg_strcasecmp(cmd, "ANALYZE") == 0)
cmdtype = PROGRESS_COMMAND_ANALYZE;
else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
@@ -552,6 +561,9 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
nulls[i + 3] = true;
}
+ if (callback)
+ callback(values, 3);
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..e4f3cd9133 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f2..5642fae0cd 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -336,4 +336,12 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+/* in commands/vacuumparallel.c */
+extern Size VacuumWorkerProgressShmemSize(void);
+extern void VacuumWorkerProgressShmemInit(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_progress_cb(Datum *values, int offset);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+
#endif /* VACUUM_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 1420288d67..dc27b8614e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_replication| SELECT s.pid,
--
2.32.0
[application/octet-stream] 0001-Expose-the-index-being-processed-in-the-vacuuming-in.patch (16.8K, ../../[email protected]/3-0001-Expose-the-index-being-processed-in-the-vacuuming-in.patch)
download | inline diff:
From 4bb70bc6dd3005d07b3c9b74d527e31b446b833f Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 17 Feb 2022 06:27:10 +0000
Subject: [PATCH 1/1] Expose the index being processed in the "vacuuming
indexes" or "cleaning up indexes" phase of a VACUUM operation.
A new view called pg_stat_progress_vacuum_index to show the indexrelid
being vacuumed or cleaned during a vacuum. The view also shows the
number of tuples removed for the index during the vacuuming indexes phase.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
---
src/backend/access/gin/ginvacuum.c | 3 ++
src/backend/access/gist/gistvacuum.c | 3 ++
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 15 ++++++++++
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +++
src/backend/catalog/system_views.sql | 41 +++++++++++++++++++++++++++
src/backend/commands/vacuumparallel.c | 12 ++++++++
src/backend/utils/adt/pgstatfuncs.c | 2 ++
src/include/commands/progress.h | 17 +++++++----
src/include/utils/backend_progress.h | 1 +
src/test/regress/expected/rules.out | 36 +++++++++++++++++++++++
12 files changed, 130 insertions(+), 6 deletions(-)
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..1d5d003780 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,7 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..23dacee52e 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -632,6 +632,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 514582972d..57dc9bc63b 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -370,6 +370,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
/*
* Get OldestXmin cutoff, which is used to determine which deleted tuples
@@ -2347,6 +2348,9 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being vacuumed in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
@@ -2358,6 +2362,10 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
*/
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+ /* Advertise we are done vacuuming indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2719,12 +2727,19 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being cleaned in non-parallel vacuum*/
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
/* See the lazy_vacuum_all_indexes comments */
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+
+ /* Advertise we are done cleaning indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
}
}
else
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c9b4964c1e..09edf49082 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1273,6 +1273,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index eaa0508c0b..5bd6953996 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1123,6 +1123,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 4 THEN 'cleaning up indexes'
WHEN 5 THEN 'truncating heap'
WHEN 6 THEN 'performing final cleanup'
+ WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
@@ -1131,6 +1133,45 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid,
+ S.datid,
+ S.datname,
+ S.indexrelid,
+ S.leader_pid,
+ CASE S.phase WHEN 2 THEN 'vacuuming indexes'
+ WHEN 4 THEN 'cleaning up indexes'
+ WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
+ END AS phase,
+ S.tuples_removed
+ FROM (
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 IN (2, 4, 7, 8) AND S.param10 > 0
+ UNION ALL
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM_PARALLEL') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ ) AS S
+ WHERE S.phase IN (2, 4, 7, 8) AND S.indexrelid > 0;
+
CREATE VIEW pg_stat_progress_cluster AS
SELECT
S.pid AS pid,
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 9b465e12cc..0c6eac8bb9 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -845,14 +845,19 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ /* Advertise the index we are cleaning or vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL);
vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL);
vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
@@ -888,6 +893,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
*/
indstats->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+ /* Advertise we are no longer vacuuming/cleaning an index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
@@ -972,6 +981,8 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL, RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -1041,6 +1052,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
error_context_stack = errcallback.previous;
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
+ pgstat_progress_end_command();
table_close(rel, ShareUpdateExclusiveLock);
FreeAccessStrategy(pvs.bstrategy);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index e9b41da6b9..4813fae40e 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -493,6 +493,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
cmdtype = PROGRESS_COMMAND_VACUUM;
callback = vacuum_progress_cb;
}
+ else if (pg_strcasecmp(cmd, "VACUUM_PARALLEL") == 0)
+ cmdtype = PROGRESS_COMMAND_VACUUM_PARALLEL;
else if (pg_strcasecmp(cmd, "ANALYZE") == 0)
cmdtype = PROGRESS_COMMAND_ANALYZE;
else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e4f3cd9133..c20466cc5f 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -27,14 +27,19 @@
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
#define PROGRESS_VACUUM_TOTAL_INDEXES 7
#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
+#define PROGRESS_VACUUM_INDEXRELID 9
+#define PROGRESS_VACUUM_TUPLES_REMOVED 10
+#define PROGRESS_VACUUM_LEADER_PID 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
-#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
-#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
-#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
-#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
-#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
-#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
+#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
+#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
+#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL 7
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL 8
/* Progress parameters for analyze */
#define PROGRESS_ANALYZE_PHASE 0
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..4651e45c40 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -23,6 +23,7 @@ typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_VACUUM_PARALLEL,
PROGRESS_COMMAND_ANALYZE,
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index dc27b8614e..c230c6a441 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1995,6 +1995,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
WHEN 4 THEN 'cleaning up indexes'::text
WHEN 5 THEN 'truncating heap'::text
WHEN 6 THEN 'performing final cleanup'::text
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
ELSE NULL::text
END AS phase,
s.param2 AS heap_blks_total,
@@ -2007,6 +2009,40 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ s.datname,
+ s.indexrelid,
+ s.leader_pid,
+ CASE s.phase
+ WHEN 2 THEN 'vacuuming indexes'::text
+ WHEN 4 THEN 'cleaning up indexes'::text
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
+ ELSE NULL::text
+ END AS phase,
+ s.tuples_removed
+ FROM ( SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))
+ WHERE ((s_1.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s_1.param10 > 0))
+ UNION ALL
+ SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM_PARALLEL'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))) s
+ WHERE ((s.phase = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s.indexrelid > 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
[application/octet-stream] 0001-Rename-index_vacuum_count-to-index_vacuum_cycle_coun.patch (2.9K, ../../[email protected]/4-0001-Rename-index_vacuum_count-to-index_vacuum_cycle_coun.patch)
download | inline diff:
From f907d2e18743502a04e4c6ef878612cbcf99bf61 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 17 Feb 2022 13:39:38 +0000
Subject: [PATCH 1/1] Rename "index_vacuum_count" to "index_vacuum_cycle_count"
in in pg_stat_pogress_vacuum.
Commit 76d8a1bfd5207d28a4e9fe98a0e1ea7c096d70aa introduces 2 new columns
to track index vacuum/cleanup progress. The columns have "index" in the
name. To make it clear what the existing "index_vacuum_count" column
refers to, which is the cycle count of the index vacuum, this change
renames the column to "index_vacuum_count"
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/catalog/system_views.sql | 2 +-
src/test/regress/expected/rules.out | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 04440dfa88..7bf31f9686 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6252,7 +6252,7 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>index_vacuum_count</structfield> <type>bigint</type>
+ <structfield>index_vacuum_cycle_count</structfield> <type>bigint</type>
</para>
<para>
Number of completed index vacuum cycles.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5bd6953996..0fa9c927e4 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1127,7 +1127,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 8 THEN 'cleaning up indexes'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
- S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+ S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_cycle_count,
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index c230c6a441..c08bcb8e74 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param2 AS heap_blks_total,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
- s.param5 AS index_vacuum_count,
+ s.param5 AS index_vacuum_cycle_count,
s.param6 AS max_dead_tuples,
s.param7 AS num_dead_tuples,
s.param8 AS indexes_total,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-02-21 19:03 ` Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-02-21 19:03 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
> The change has been broken up as 3 separate patches.
> 0007-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch - Introduces 2 new columns to pg_stat_progress_vacuum, indexes_total and indexes_processed. These 2 columns will provide progress on the index vacuuming/cleanup.
> 0001-Expose-the-index-being-processed-in-the-vacuuming-in.patch - Introduces a new view called pg_stat_prgoress_vacuum_index. This view tracks the index being vacuumed/cleaned and the total number of index tuples removed.
> 0001-Rename-index_vacuum_count-to-index_vacuum_cycle_coun.patch - Renames the existing index_vacuum_count to index_vacuum_cycle_count in pg_stat_progress_vacuum. Due to the other changes, it makes sense to include "cycle" in the column name to be crystal clear that the column refers to the index cycle count.
> Thanks
Sending again with patch files renamed to ensure correct apply order.
On 2/10/22, 1:39 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
Attached is the latest version of the patch to deal with the changes in the recent commit aa64f23b02924724eafbd9eadbf26d85df30a12b
On 2/1/22, 2:32 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
After speaking with Nathan offline, A few changes have been made to the patch.
As mentioned earlier in the thread, tracking how many indexes are processed in PARALLEL vacuum mode is not very straightforward since only the workers or leader process have ability to inspect the Vacuum shared parallel state.
The latest version of the patch introduces a shared memory to track indexes vacuumed/cleaned by each worker ( or leader ) in a PARALLEL vacuum. In order to present this data in the pg_stat_progress_vacuum view, the value of the new column "indexes_processed" is retrieved from shared memory by pg_stat_get_progress_info. For non-parallel vacuums, the value of "indexes_processed" is retrieved from the backend progress array directly.
The patch also includes the changes to implement the new view pg_stat_progress_vacuum_index which exposes the index being vacuumed/cleaned up.
postgres=# \d+ pg_stat_progress_vacuum ;
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default | Storage | Description
--------------------+---------+-----------+----------+---------+----------+-------------
pid | integer | | | | plain |
datid | oid | | | | plain |
datname | name | | | | plain |
relid | oid | | | | plain |
phase | text | | | | extended |
heap_blks_total | bigint | | | | plain |
heap_blks_scanned | bigint | | | | plain |
heap_blks_vacuumed | bigint | | | | plain |
index_vacuum_count | bigint | | | | plain |
max_dead_tuples | bigint | | | | plain |
num_dead_tuples | bigint | | | | plain |
indexes_total | bigint | | | | plain | <<<-- new column
indexes_processed | bigint | | | | plain | <<<-- new column
<<<--- new view --->>>
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
leader_pid | bigint | | |
phase | text | | |
tuples_removed | bigint | | |
On 1/26/22, 8:07 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
Attached is the latest patch and associated documentation.
This version addresses the index_ordinal_position column confusion. Rather than displaying the index position, the pg_stat_progress_vacuum view now has 2 new column(s):
index_total - this column will show the total number of indexes to be vacuumed
index_complete_count - this column will show the total number of indexes processed so far. In order to deal with the parallel vacuums, the parallel_workers ( planned workers ) value had to be exposed and each backends performing an index vacuum/cleanup in parallel had to advertise the number of indexes it vacuumed/cleaned. The # of indexes vacuumed for the parallel cleanup can then be derived the pg_stat_progress_vacuum view.
postgres=# \d pg_stat_progress_vacuum
View "pg_catalog.pg_stat_progress_vacuum"
Column | Type | Collation | Nullable | Default
----------------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
relid | oid | | |
phase | text | | |
heap_blks_total | bigint | | |
heap_blks_scanned | bigint | | |
heap_blks_vacuumed | bigint | | |
index_vacuum_count | bigint | | |
max_dead_tuples | bigint | | |
num_dead_tuples | bigint | | |
index_total | bigint | | |. <<<---------------------
index_complete_count | numeric | | |. <<<---------------------
The pg_stat_progress_vacuum_index view includes:
Indexrelid - the currently vacuumed index
Leader_pid - the pid of the leader process. NULL if the process is the leader or vacuum is not parallel
tuples_removed - the amount of indexes tuples removed. The user can use this column to see that the index vacuum has movement.
postgres=# \d pg_stat_progress_vacuum_index
View "pg_catalog.pg_stat_progress_vacuum_index"
Column | Type | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
pid | integer | | |
datid | oid | | |
datname | name | | |
indexrelid | bigint | | |
phase | text | | |
leader_pid | bigint | | |
tuples_removed | bigint | | |
On 1/12/22, 9:52 PM, "Imseih (AWS), Sami" <[email protected]> wrote:
On 1/12/22, 1:28 PM, "Bossart, Nathan" <[email protected]> wrote:
On 1/11/22, 11:46 PM, "Masahiko Sawada" <[email protected]> wrote:
> Regarding the new pg_stat_progress_vacuum_index view, why do we need
> to have a separate view? Users will have to check two views. If this
> view is expected to be used together with and joined to
> pg_stat_progress_vacuum, why don't we provide one view that has full
> information from the beginning? Especially, I think it's not useful
> that the total number of indexes to vacuum (num_indexes_to_vacuum
> column) and the current number of indexes that have been vacuumed
> (index_ordinal_position column) are shown in separate views.
> I suppose we could add all of the new columns to
> pg_stat_progress_vacuum and just set columns to NULL as appropriate.
> But is that really better than having a separate view?
To add, since a vacuum can utilize parallel worker processes + the main vacuum process to perform index vacuuming, it made sense to separate the backends doing index vacuum/cleanup in a separate view.
Besides what Nathan suggested, the only other clean option I can think of is to perhaps create a json column in pg_stat_progress_vacuum which will include all the new fields. My concern with this approach is that it will make usability, to flatten the json, difficult for users.
> Also, I’m not sure how useful index_tuples_removed is; what can we
> infer from this value (without a total number)?
> I think the idea was that you can compare it against max_dead_tuples
> and num_dead_tuples to get an estimate of the current cycle progress.
> Otherwise, it just shows that progress is being made.
The main purpose is to really show that the "index vacuum" phase is actually making progress. Note that for certain types of indexes, i.e. GIN/GIST the number of tuples_removed will end up exceeding the number of num_dead_tuples.
Nathan
[0] https://postgr.es/m/7874FB21-FAA5-49BD-8386-2866552656C7%40amazon.com
Attachments:
[application/octet-stream] 0011-Rename-index_vacuum_count-to-index_vacuum_cycle_coun.patch (2.9K, ../../[email protected]/2-0011-Rename-index_vacuum_count-to-index_vacuum_cycle_coun.patch)
download | inline diff:
From f907d2e18743502a04e4c6ef878612cbcf99bf61 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 17 Feb 2022 13:39:38 +0000
Subject: [PATCH 1/1] Rename "index_vacuum_count" to "index_vacuum_cycle_count"
in in pg_stat_pogress_vacuum.
Commit 76d8a1bfd5207d28a4e9fe98a0e1ea7c096d70aa introduces 2 new columns
to track index vacuum/cleanup progress. The columns have "index" in the
name. To make it clear what the existing "index_vacuum_count" column
refers to, which is the cycle count of the index vacuum, this change
renames the column to "index_vacuum_count"
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/catalog/system_views.sql | 2 +-
src/test/regress/expected/rules.out | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 04440dfa88..7bf31f9686 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6252,7 +6252,7 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>index_vacuum_count</structfield> <type>bigint</type>
+ <structfield>index_vacuum_cycle_count</structfield> <type>bigint</type>
</para>
<para>
Number of completed index vacuum cycles.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5bd6953996..0fa9c927e4 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1127,7 +1127,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 8 THEN 'cleaning up indexes'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
- S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+ S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_cycle_count,
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index c230c6a441..c08bcb8e74 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param2 AS heap_blks_total,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
- s.param5 AS index_vacuum_count,
+ s.param5 AS index_vacuum_cycle_count,
s.param6 AS max_dead_tuples,
s.param7 AS num_dead_tuples,
s.param8 AS indexes_total,
--
2.32.0
[application/octet-stream] 0010-Expose-the-index-being-processed-in-the-vacuuming-in.patch (21.6K, ../../[email protected]/3-0010-Expose-the-index-being-processed-in-the-vacuuming-in.patch)
download | inline diff:
From 6d38d09c72e917db1c078d93a5278efab5c771c2 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 17 Feb 2022 16:25:12 +0000
Subject: [PATCH 1/1] Expose the index being processed in the "vacuuming
indexes" or "cleaning up indexes" phase of a VACUUM operation.
A new view called pg_stat_progress_vacuum_index to show the indexrelid
being vacuumed or cleaned during a vacuum. The view also shows the
number of tuples removed for the index during the vacuuming indexes phase.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 104 +++++++++++++++++++++++++-
src/backend/access/gin/ginvacuum.c | 3 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 15 ++++
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 41 ++++++++++
src/backend/commands/vacuumparallel.c | 12 +++
src/backend/utils/adt/pgstatfuncs.c | 2 +
src/include/commands/progress.h | 17 +++--
src/include/utils/backend_progress.h | 1 +
src/test/regress/expected/rules.out | 36 +++++++++
13 files changed, 233 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 04440dfa88..85369acb39 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -379,6 +379,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_progress_vacuum_index</structname><indexterm><primary>pg_stat_progress_vacuum_index</primary></indexterm></entry>
+ <entry>One row for each backend processing an index during a
+ <command>VACUUM</command>, showing current progress.
+ See <xref linkend='vacuum-progress-reporting'/>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_cluster</structname><indexterm><primary>pg_stat_progress_cluster</primary></indexterm></entry>
<entry>One row for each backend running
@@ -6144,7 +6152,10 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Whenever <command>VACUUM</command> is running, the
<structname>pg_stat_progress_vacuum</structname> view will contain
one row for each backend (including autovacuum worker processes) that is
- currently vacuuming. The tables below describe the information
+ currently vacuuming. The <structname>pg_stat_progress_vacuum_index</structname>
+ view will also contain one row for each backend processing an index
+ during the <command>VACUUM</command>.
+ The tables below describe the information
that will be reported and provide information about how to interpret it.
Progress for <command>VACUUM FULL</command> commands is reported via
<structname>pg_stat_progress_cluster</structname>
@@ -6304,6 +6315,97 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</tgroup>
</table>
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
<table id="vacuum-phases">
<title>VACUUM Phases</title>
<tgroup cols="2">
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..1d5d003780 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,7 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..23dacee52e 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -632,6 +632,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 514582972d..57dc9bc63b 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -370,6 +370,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
/*
* Get OldestXmin cutoff, which is used to determine which deleted tuples
@@ -2347,6 +2348,9 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being vacuumed in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
@@ -2358,6 +2362,10 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
*/
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+ /* Advertise we are done vacuuming indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2719,12 +2727,19 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being cleaned in non-parallel vacuum*/
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
/* See the lazy_vacuum_all_indexes comments */
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+
+ /* Advertise we are done cleaning indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
}
}
else
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c9b4964c1e..09edf49082 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1273,6 +1273,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index eaa0508c0b..5bd6953996 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1123,6 +1123,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 4 THEN 'cleaning up indexes'
WHEN 5 THEN 'truncating heap'
WHEN 6 THEN 'performing final cleanup'
+ WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
@@ -1131,6 +1133,45 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid,
+ S.datid,
+ S.datname,
+ S.indexrelid,
+ S.leader_pid,
+ CASE S.phase WHEN 2 THEN 'vacuuming indexes'
+ WHEN 4 THEN 'cleaning up indexes'
+ WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
+ END AS phase,
+ S.tuples_removed
+ FROM (
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 IN (2, 4, 7, 8) AND S.param10 > 0
+ UNION ALL
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM_PARALLEL') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ ) AS S
+ WHERE S.phase IN (2, 4, 7, 8) AND S.indexrelid > 0;
+
CREATE VIEW pg_stat_progress_cluster AS
SELECT
S.pid AS pid,
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 9b465e12cc..0c6eac8bb9 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -845,14 +845,19 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ /* Advertise the index we are cleaning or vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL);
vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL);
vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
@@ -888,6 +893,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
*/
indstats->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+ /* Advertise we are no longer vacuuming/cleaning an index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
@@ -972,6 +981,8 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL, RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -1041,6 +1052,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
error_context_stack = errcallback.previous;
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
+ pgstat_progress_end_command();
table_close(rel, ShareUpdateExclusiveLock);
FreeAccessStrategy(pvs.bstrategy);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index e9b41da6b9..4813fae40e 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -493,6 +493,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
cmdtype = PROGRESS_COMMAND_VACUUM;
callback = vacuum_progress_cb;
}
+ else if (pg_strcasecmp(cmd, "VACUUM_PARALLEL") == 0)
+ cmdtype = PROGRESS_COMMAND_VACUUM_PARALLEL;
else if (pg_strcasecmp(cmd, "ANALYZE") == 0)
cmdtype = PROGRESS_COMMAND_ANALYZE;
else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e4f3cd9133..c20466cc5f 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -27,14 +27,19 @@
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
#define PROGRESS_VACUUM_TOTAL_INDEXES 7
#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
+#define PROGRESS_VACUUM_INDEXRELID 9
+#define PROGRESS_VACUUM_TUPLES_REMOVED 10
+#define PROGRESS_VACUUM_LEADER_PID 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
-#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
-#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
-#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
-#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
-#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
-#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
+#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
+#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
+#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL 7
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL 8
/* Progress parameters for analyze */
#define PROGRESS_ANALYZE_PHASE 0
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..4651e45c40 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -23,6 +23,7 @@ typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_VACUUM_PARALLEL,
PROGRESS_COMMAND_ANALYZE,
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index dc27b8614e..c230c6a441 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1995,6 +1995,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
WHEN 4 THEN 'cleaning up indexes'::text
WHEN 5 THEN 'truncating heap'::text
WHEN 6 THEN 'performing final cleanup'::text
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
ELSE NULL::text
END AS phase,
s.param2 AS heap_blks_total,
@@ -2007,6 +2009,40 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ s.datname,
+ s.indexrelid,
+ s.leader_pid,
+ CASE s.phase
+ WHEN 2 THEN 'vacuuming indexes'::text
+ WHEN 4 THEN 'cleaning up indexes'::text
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
+ ELSE NULL::text
+ END AS phase,
+ s.tuples_removed
+ FROM ( SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))
+ WHERE ((s_1.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s_1.param10 > 0))
+ UNION ALL
+ SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM_PARALLEL'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))) s
+ WHERE ((s.phase = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s.indexrelid > 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
[application/octet-stream] 0009-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch (17.3K, ../../[email protected]/4-0009-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch)
download | inline diff:
From 76d8a1bfd5207d28a4e9fe98a0e1ea7c096d70aa Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 17 Feb 2022 04:21:04 +0000
Subject: [PATCH 1/1] Expose progress for the "vacuuming indexes" and "cleaning
up indexes" phase of a VACUUM operation.
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 22 +++
src/backend/access/heap/vacuumlazy.c | 207 +++++++++++++++++++++++++-
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 7 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/adt/pgstatfuncs.c | 12 ++
src/include/commands/progress.h | 2 +
src/include/commands/vacuum.h | 8 +
src/test/regress/expected/rules.out | 4 +-
9 files changed, 261 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index bf7625d988..04440dfa88 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6278,6 +6278,28 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase.
+ At the start of an index vacuum cycle, this value is set to <literal>0</literal>.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 242511a235..1198677bc0 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -245,6 +246,26 @@ typedef struct LVSavedErrInfo
VacErrPhase phase;
} LVSavedErrInfo;
+/*
+ * Structs for tracking shared Progress information
+ * amongst worker ( and leader ) processes of a vacuum.
+ */
+typedef struct VacOneWorkerProgressInfo
+{
+ int leader_pid;
+ int indexes_processed;
+} VacOneWorkerProgressInfo;
+
+typedef struct VacWorkerProgressInfo
+{
+ int num_vacuums; /* number of active VACUUMS with parallel workers */
+ int max_vacuums; /* max number of VACUUMS with parallel workers */
+ slock_t mutex;
+ VacOneWorkerProgressInfo vacuums[FLEXIBLE_ARRAY_MEMBER];
+} VacWorkerProgressInfo;
+
+static VacWorkerProgressInfo *vacworkerprogress;
+
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
@@ -420,6 +441,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2328,6 +2351,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /*
+ * For the non-parallel variant of a vacuum, the array position
+ * of the index determines how many indexes are processed so far.
+ * Add 1 to the posititon as vacrel->nindexes is a 0-based array.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2338,9 +2368,20 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
- vacrel->num_index_scans);
+ /*
+ * parallel_vacuum_bulkdel_all_indexes will call vacuum_worker_update
+ * which updates shared memory for the index progress. To ensure shared
+ * memory cleanup, do the work with PG_ENSURE_ERROR_CLEANUP.
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs,
+ vacrel->old_live_tuples,
+ vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2350,6 +2391,12 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
allindexes = false;
}
+ /*
+ * We're done with index vacuuming.
+ * Set the total number of indexes completed as the total number of indexes
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, vacrel->nindexes) ;
+
/*
* We delete all LP_DEAD items from the first heap pass in all indexes on
* each call here (except calls where we choose to do the failsafe). This
@@ -2675,15 +2722,34 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /* See the lazy_vacuum_all_indexes comments */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ /*
+ * Outsource everything to parallel variant
+ *
+ * See the lazy_vacuum_all_indexes comments
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
vacrel->num_index_scans,
(vacrel->scanned_pages < vacrel->rel_pages));
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
}
+
+ /*
+ * We're done with index cleanup.
+ * Set the total number of indexes completed as the total number of indexes
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, vacrel->nindexes) ;
+
}
/*
@@ -3464,3 +3530,134 @@ restore_vacuum_error_info(LVRelState *vacrel,
vacrel->offnum = saved_vacrel->offnum;
vacrel->phase = saved_vacrel->phase;
}
+
+void
+vacuum_worker_end(int leader_pid)
+{
+ SpinLockAcquire(&vacworkerprogress->mutex);
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ VacOneWorkerProgressInfo *vac = &vacworkerprogress->vacuums[i];
+
+ if (vac->leader_pid == leader_pid)
+ {
+ *vac = vacworkerprogress->vacuums[vacworkerprogress->num_vacuums - 1];
+ vacworkerprogress->num_vacuums--;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ break;
+ }
+ }
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
+
+/*
+ * vacuum_worker_end wrapped as an on_shmem_exit callback function
+ */
+void
+vacuum_worker_end_callback(int code, Datum arg)
+{
+ vacuum_worker_end(DatumGetInt32(arg));
+}
+
+/*
+ * vacuum_worker_update sets the number of indexes processed so far
+ * in a parallel vacuum.
+ */
+void
+vacuum_worker_update(int leader_pid)
+{
+ VacOneWorkerProgressInfo *vac;
+
+ SpinLockAcquire(&vacworkerprogress->mutex);
+
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ int next_leader_pid;
+
+ vac = &vacworkerprogress->vacuums[i];
+
+ next_leader_pid = vac->leader_pid;
+
+ if (next_leader_pid == leader_pid)
+ {
+ vac->indexes_processed++;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ return;
+ }
+ }
+
+ if (vacworkerprogress->num_vacuums >= vacworkerprogress->max_vacuums)
+ {
+ SpinLockRelease(&vacworkerprogress->mutex);
+ elog(ERROR, "out of vacuum worker progress slots");
+ }
+
+ vac = &vacworkerprogress->vacuums[vacworkerprogress->num_vacuums];
+ vac->leader_pid = leader_pid;
+ vac->indexes_processed = 1;
+ vacworkerprogress->num_vacuums++;
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
+
+/*
+ * vacuum_progress_cb updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+vacuum_progress_cb(Datum *values, int offset)
+{
+ VacOneWorkerProgressInfo *vac;
+ int leader_pid = values[0];
+
+ /* If we are vacuuming in parallel, set the number of indexes vacuumed
+ * from the shared memory counter.
+ * */
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ int next_leader_pid;
+
+ vac = &vacworkerprogress->vacuums[i];
+
+ next_leader_pid = vac->leader_pid;
+
+ if (next_leader_pid == leader_pid)
+ values[PROGRESS_VACUUM_INDEXES_COMPLETED + offset] = vac->indexes_processed;
+ }
+}
+
+/*
+ * VacuumWorkerProgressShmemSize --- report amount of shared memory space needed
+ */
+Size
+VacuumWorkerProgressShmemSize(void)
+{
+ Size size;
+
+ size = offsetof(VacWorkerProgressInfo, vacuums);
+ size = add_size(size, mul_size(GetMaxBackends(), sizeof(VacOneWorkerProgressInfo)));
+ return size;
+}
+
+/*
+ * VacuumWorkerProgressShmemInit --- initialize this module's shared memory
+ */
+void
+VacuumWorkerProgressShmemInit(void)
+{
+ bool found;
+
+ vacworkerprogress = (VacWorkerProgressInfo *) ShmemInitStruct("Vacuum Worker Progress Stats",
+ VacuumWorkerProgressShmemSize(),
+ &found);
+
+ if (!IsUnderPostmaster)
+ {
+ /* Initialize shared memory area */
+ Assert(!found);
+
+ vacworkerprogress->max_vacuums = GetMaxBackends();
+ SpinLockInit(&vacworkerprogress->mutex);
+ }
+ else
+ Assert(found);
+}
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3cb69b1f87..eaa0508c0b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1126,7 +1126,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..9b465e12cc 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -844,9 +849,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index cd4ebe2fc5..a4bd6a14a3 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -24,6 +24,7 @@
#include "access/twophase.h"
#include "access/xlogrecovery.h"
#include "commands/async.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -145,6 +146,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, VacuumWorkerProgressShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -296,6 +298,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ VacuumWorkerProgressShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 15cb17ace4..3d90b23ce4 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,7 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/vacuum.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -452,6 +453,10 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
/*
* Returns command progress information for the named command.
+ *
+ * A command type can optionally define a callback function
+ * which will derive Datum values rather than use values
+ * directly from the backends progress array.
*/
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
@@ -466,6 +471,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
MemoryContext per_query_ctx;
MemoryContext oldcontext;
+ void (*callback)(Datum *, int) = NULL;
/* check to see if caller supports us returning a tuplestore */
if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
@@ -483,7 +489,10 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ {
cmdtype = PROGRESS_COMMAND_VACUUM;
+ callback = vacuum_progress_cb;
+ }
else if (pg_strcasecmp(cmd, "ANALYZE") == 0)
cmdtype = PROGRESS_COMMAND_ANALYZE;
else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
@@ -552,6 +561,9 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
nulls[i + 3] = true;
}
+ if (callback)
+ callback(values, 3);
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..e4f3cd9133 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f2..5642fae0cd 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -336,4 +336,12 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+/* in commands/vacuumparallel.c */
+extern Size VacuumWorkerProgressShmemSize(void);
+extern void VacuumWorkerProgressShmemInit(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_progress_cb(Datum *values, int offset);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+
#endif /* VACUUM_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 1420288d67..dc27b8614e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_replication| SELECT s.pid,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-02-21 19:09 ` Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Nathan Bossart @ 2022-02-21 19:09 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
On Mon, Feb 21, 2022 at 07:03:39PM +0000, Imseih (AWS), Sami wrote:
> Sending again with patch files renamed to ensure correct apply order.
I haven't had a chance to test this too much, but I did look through the
patch set and have a couple of small comments.
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase.
+ At the start of an index vacuum cycle, this value is set to <literal>0</literal>.
+ </para></entry>
+ </row>
Will these be set to 0 for failsafe vacuums and vacuums with INDEX_CLEANUP
turned off?
+typedef struct VacWorkerProgressInfo
+{
+ int num_vacuums; /* number of active VACUUMS with parallel workers */
+ int max_vacuums; /* max number of VACUUMS with parallel workers */
+ slock_t mutex;
+ VacOneWorkerProgressInfo vacuums[FLEXIBLE_ARRAY_MEMBER];
+} VacWorkerProgressInfo;
max_vacuums appears to just be a local copy of MaxBackends. Does this
information really need to be stored here? Also, is there a strong reason
for using a spinlock instead of an LWLock?
+void
+vacuum_worker_end(int leader_pid)
+{
+ SpinLockAcquire(&vacworkerprogress->mutex);
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ VacOneWorkerProgressInfo *vac = &vacworkerprogress->vacuums[i];
+
+ if (vac->leader_pid == leader_pid)
+ {
+ *vac = vacworkerprogress->vacuums[vacworkerprogress->num_vacuums - 1];
+ vacworkerprogress->num_vacuums--;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ break;
+ }
+ }
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
I see this loop pattern in a couple of places, and it makes me wonder if
this information would fit more naturally in a hash table.
+ if (callback)
+ callback(values, 3);
Why does this need to be set up as a callback function? Could we just call
the function if cmdtype == PROGRESS_COMMAND_VACUUM? ISTM that is pretty
much all this is doing.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
@ 2022-02-23 18:02 ` Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-02-23 18:02 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; Justin Pryzby <[email protected]>; Peter Geoghegan <[email protected]>; pgsql-hackers
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase
+ of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes processed in the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal> phase.
+ At the start of an index vacuum cycle, this value is set to <literal>0</literal>.
+ </para></entry>
+ </row>
> Will these be set to 0 for failsafe vacuums and vacuums with INDEX_CLEANUP
> turned off?
If the failsafe kicks in midway through a vacuum, the number indexes_total will not be reset to 0. If INDEX_CLEANUP is turned off, then the value will be 0 at the start of the vacuum.
+typedef struct VacWorkerProgressInfo
+{
+ int num_vacuums; /* number of active VACUUMS with parallel workers */
+ int max_vacuums; /* max number of VACUUMS with parallel workers */
+ slock_t mutex;
+ VacOneWorkerProgressInfo vacuums[FLEXIBLE_ARRAY_MEMBER];
+} VacWorkerProgressInfo;
> max_vacuums appears to just be a local copy of MaxBackends. Does this
> information really need to be stored here? Also, is there a strong reason
> for using a spinlock instead of an LWLock?
First, The BTVacInfo code in backend/access/nbtree/nbtutils.c inspired this, so I wanted to follow this pattern. With that said, I do see max_vacuums being redundant here, and I am inclined to replace it with a MaxBackends() call.
Second, There is no strong reason to use spinlock here except I incorrectly assumed it will be better for this case. After reading more about this and reading up src/backend/storage/lmgr/README, an LWLock will be better.
+void
+vacuum_worker_end(int leader_pid)
+{
+ SpinLockAcquire(&vacworkerprogress->mutex);
+ for (int i = 0; i < vacworkerprogress->num_vacuums; i++)
+ {
+ VacOneWorkerProgressInfo *vac = &vacworkerprogress->vacuums[i];
+
+ if (vac->leader_pid == leader_pid)
+ {
+ *vac = vacworkerprogress->vacuums[vacworkerprogress->num_vacuums - 1];
+ vacworkerprogress->num_vacuums--;
+ SpinLockRelease(&vacworkerprogress->mutex);
+ break;
+ }
+ }
+ SpinLockRelease(&vacworkerprogress->mutex);
+}
> I see this loop pattern in a couple of places, and it makes me wonder if
> this information would fit more naturally in a hash table.
Followed the pattern in backend/access/nbtree/nbtutils.c for this as well. Using dynahash may make sense here if it simplifies the code. Will look.
+ if (callback)
+ callback(values, 3);
> Why does this need to be set up as a callback function? Could we just call
> the function if cmdtype == PROGRESS_COMMAND_VACUUM? ISTM that is pretty
> much all this is doing.
The intention will be for the caller to set the callback early on in the function using the existing " if (pg_strcasecmp(cmd, "VACUUM") == 0), etc." statement. This way we avoid having to add another if/else block before tuplestore_putvalues is called.
--
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-02-23 18:41 ` Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Peter Geoghegan @ 2022-02-23 18:41 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Wed, Feb 23, 2022 at 10:02 AM Imseih (AWS), Sami <[email protected]> wrote:
> If the failsafe kicks in midway through a vacuum, the number indexes_total will not be reset to 0. If INDEX_CLEANUP is turned off, then the value will be 0 at the start of the vacuum.
The way that this works with num_index_scans is that we "round up"
when there has been non-zero work in lazy_vacuum_all_indexes(), but
not if the precheck in lazy_vacuum_all_indexes() fails. That seems
like a good model to generalize from here. Note that this makes
INDEX_CLEANUP=off affect num_index_scans in much the same way as a
VACUUM where the failsafe kicks in very early, during the initial heap
pass. That is, if the failsafe kicks in before we reach lazy_vacuum()
for the first time (which is not unlikely), or even in the
lazy_vacuum_all_indexes() precheck, then num_index_scans will remain
at 0, just like INDEX_CLEANUP=off.
The actual failsafe WARNING shows num_index_scans, possibly before it
gets incremented one last time (by "rounding up"). So it's reasonably
clear how this all works from that context (assuming that the
autovacuum logging stuff, which reports num_index_scans, outputs a
report for a table where the failsafe kicked in).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
@ 2022-02-25 19:52 ` Nathan Bossart <[email protected]>
2022-02-27 17:16 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Nathan Bossart @ 2022-02-25 19:52 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: Imseih (AWS), Sami <[email protected]>; Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Wed, Feb 23, 2022 at 10:41:36AM -0800, Peter Geoghegan wrote:
> On Wed, Feb 23, 2022 at 10:02 AM Imseih (AWS), Sami <[email protected]> wrote:
>> If the failsafe kicks in midway through a vacuum, the number indexes_total will not be reset to 0. If INDEX_CLEANUP is turned off, then the value will be 0 at the start of the vacuum.
>
> The way that this works with num_index_scans is that we "round up"
> when there has been non-zero work in lazy_vacuum_all_indexes(), but
> not if the precheck in lazy_vacuum_all_indexes() fails. That seems
> like a good model to generalize from here. Note that this makes
> INDEX_CLEANUP=off affect num_index_scans in much the same way as a
> VACUUM where the failsafe kicks in very early, during the initial heap
> pass. That is, if the failsafe kicks in before we reach lazy_vacuum()
> for the first time (which is not unlikely), or even in the
> lazy_vacuum_all_indexes() precheck, then num_index_scans will remain
> at 0, just like INDEX_CLEANUP=off.
>
> The actual failsafe WARNING shows num_index_scans, possibly before it
> gets incremented one last time (by "rounding up"). So it's reasonably
> clear how this all works from that context (assuming that the
> autovacuum logging stuff, which reports num_index_scans, outputs a
> report for a table where the failsafe kicked in).
I am confused. If failsafe kicks in during the middle of a vacuum, I
(perhaps naively) would expect indexes_total and indexes_processed to go to
zero, and I'd expect to no longer see the "vacuuming indexes" and "cleaning
up indexes" phases. Otherwise, how would I know that we are now skipping
indexes? Of course, you won't have any historical context about the index
work done before failsafe kicked in, but IMO it is misleading to still
include it in the progress view.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
@ 2022-02-27 17:16 ` Imseih (AWS), Sami <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-02-27 17:16 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> On Wed, Feb 23, 2022 at 10:02 AM Imseih (AWS), Sami <[email protected]> wrote:
>> If the failsafe kicks in midway through a vacuum, the number indexes_total will not be reset to 0. If INDEX_CLEANUP is turned off, then the value will be 0 at the start of the vacuum.
>
> The way that this works with num_index_scans is that we "round up"
> when there has been non-zero work in lazy_vacuum_all_indexes(), but
> not if the precheck in lazy_vacuum_all_indexes() fails. That seems
> like a good model to generalize from here. Note that this makes
> INDEX_CLEANUP=off affect num_index_scans in much the same way as a
> VACUUM where the failsafe kicks in very early, during the initial heap
> pass. That is, if the failsafe kicks in before we reach lazy_vacuum()
> for the first time (which is not unlikely), or even in the
> lazy_vacuum_all_indexes() precheck, then num_index_scans will remain
> at 0, just like INDEX_CLEANUP=off.
>
> The actual failsafe WARNING shows num_index_scans, possibly before it
> gets incremented one last time (by "rounding up"). So it's reasonably
> clear how this all works from that context (assuming that the
> autovacuum logging stuff, which reports num_index_scans, outputs a
> report for a table where the failsafe kicked in).
> I am confused. If failsafe kicks in during the middle of a vacuum, I
> (perhaps naively) would expect indexes_total and indexes_processed to go to
> zero, and I'd expect to no longer see the "vacuuming indexes" and "cleaning
> up indexes" phases. Otherwise, how would I know that we are now skipping
> indexes? Of course, you won't have any historical context about the index
> work done before failsafe kicked in, but IMO it is misleading to still
> include it in the progress view.
Failsafe occurring in the middle of a vacuum and resetting "indexes_total" to 0 will be misleading. I am thinking that it is a better idea to expose only one column "indexes_remaining".
If index_cleanup is set to OFF, the values of indexes_remaining will be 0 at the start of the vacuum.
If failsafe kicks in during a vacuum in-progress, "indexes_remaining" will be calculated to 0.
This approach will provide a progress based on how many indexes remaining with no ambiguity.
--
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
@ 2022-03-03 05:08 ` Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-03 05:08 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Masahiko Sawada <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> >> If the failsafe kicks in midway through a vacuum, the number indexes_total will not be reset to 0. If INDEX_CLEANUP is turned off, then the value will be 0 at the start of the vacuum.
> >
> > The way that this works with num_index_scans is that we "round up"
> > when there has been non-zero work in lazy_vacuum_all_indexes(), but
> > not if the precheck in lazy_vacuum_all_indexes() fails. That seems
> > like a good model to generalize from here. Note that this makes
> > INDEX_CLEANUP=off affect num_index_scans in much the same way as a
> > VACUUM where the failsafe kicks in very early, during the initial heap
> > pass. That is, if the failsafe kicks in before we reach lazy_vacuum()
> > for the first time (which is not unlikely), or even in the
> > lazy_vacuum_all_indexes() precheck, then num_index_scans will remain
> > at 0, just like INDEX_CLEANUP=off.
> >
> > The actual failsafe WARNING shows num_index_scans, possibly before it
> > gets incremented one last time (by "rounding up"). So it's reasonably
> > clear how this all works from that context (assuming that the
> > autovacuum logging stuff, which reports num_index_scans, outputs a
> > report for a table where the failsafe kicked in).
> I am confused. If failsafe kicks in during the middle of a vacuum, I
> (perhaps naively) would expect indexes_total and indexes_processed to go to
> zero, and I'd expect to no longer see the "vacuuming indexes" and "cleaning
> up indexes" phases. Otherwise, how would I know that we are now skipping
> indexes? Of course, you won't have any historical context about the index
> work done before failsafe kicked in, but IMO it is misleading to still
> include it in the progress view.
After speaking with Nathan offline, the best forward is to reset indexes_total and indexes_processed to 0 after the start of "vacuuming indexes" or "cleaning up indexes" phase.
Also, if failsafe is triggered midway through a vacuum, the values for both indexes_total and indexes_processed is (re)set to 0.
Revision of the patch is attached.
Below is a test that shows the output.
-[ RECORD 1 ]------+------------------
pid | 4360
datid | 5
datname | postgres
relid | 16399
phase | vacuuming indexes
heap_blks_total | 401092
heap_blks_scanned | 211798
heap_blks_vacuumed | 158847
index_vacuum_count | 3
max_dead_tuples | 1747625
num_dead_tuples | 1747366
indexes_total | 8 <<<<--- index_vacuum_count is 3, indexes_total is 8 and indexes_processed so far is 1
indexes_processed | 1
-[ RECORD 1 ]------+--------------
pid | 4360
datid | 5
datname | postgres
relid | 16399
phase | scanning heap
heap_blks_total | 401092
heap_blks_scanned | 234590
heap_blks_vacuumed | 211797
index_vacuum_count | 4
max_dead_tuples | 1747625
num_dead_tuples | 752136
indexes_total | 0 <<<<--- index_vacuum_count is 4 and not in an index phase. indexes_total is 0 and indexes_processed so far is 0
indexes_processed | 0
-[ RECORD 1 ]------+------------------
pid | 4360
datid | 5
datname | postgres
relid | 16399
phase | vacuuming indexes
heap_blks_total | 401092
heap_blks_scanned | 264748
heap_blks_vacuumed | 211797
index_vacuum_count | 4
max_dead_tuples | 1747625
num_dead_tuples | 1747350
indexes_total | 8
indexes_processed | 6 <<<<--- index_vacuum_count is 4, indexes_total is 8 and indexes_processed so far is 6
--
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v2-0009-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch (17.6K, ../../[email protected]/2-v2-0009-Expose-progress-for-the-vacuuming-indexes-and-cleani.patch)
download | inline diff:
From 996254bdbe792bbf1cabb914f3eab883978911a0 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 3 Mar 2022 04:37:48 +0000
Subject: [PATCH 1/1] Expose progress for the "vacuuming indexes" and "cleaning
up indexes" phase of a VACUUM operation.
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 24 ++++
src/backend/access/heap/vacuumlazy.c | 156 ++++++++++++++++++++++-
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 7 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/utils/adt/pgstatfuncs.c | 16 ++-
src/include/commands/progress.h | 5 +
src/include/commands/vacuum.h | 7 +
src/test/regress/expected/rules.out | 4 +-
10 files changed, 215 insertions(+), 10 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 9fb62fec8e..140fbd5fb9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6227,6 +6227,30 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes already processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 40101e0cb8..e9c081ada6 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -244,6 +245,16 @@ typedef struct LVSavedErrInfo
VacErrPhase phase;
} LVSavedErrInfo;
+/*
+ * Structs for tracking shared Progress information
+ * amongst worker ( and leader ) processes of a vacuum.
+ */
+typedef struct VacProgressEntry
+{
+ pid_t leader_pid;
+ int indexes_total;
+ int indexes_processed;
+} VacProgressEntry;
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
@@ -290,6 +301,7 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
+static HTAB *VacuumWorkerProgressHash;
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
@@ -2305,9 +2317,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/* Report that we are now vacuuming indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
+ int indexes_completed = 0;
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
@@ -2317,6 +2333,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2327,9 +2345,22 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
- vacrel->num_index_scans);
+ /*
+ * Outsource everything to parallel variant.
+ *
+ * * parallel_vacuum_bulkdel_all_indexes will call vacuum_worker_update
+ * which updates shared memory for the index progress. To ensure shared
+ * memory cleanup, do the work with PG_ENSURE_ERROR_CLEANUP.
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs,
+ vacrel->old_live_tuples,
+ vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2339,6 +2370,10 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
allindexes = false;
}
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
/*
* We delete all LP_DEAD items from the first heap pass in all indexes on
* each call here (except calls where we choose to do the failsafe). This
@@ -2617,6 +2652,8 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2649,12 +2686,15 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
/* Report that we are now cleaning up indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
+ /* Advertise the number of indexes we are cleaning up */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
double reltuples = vacrel->new_rel_tuples;
bool estimated_count =
vacrel->scanned_pages < vacrel->rel_pages;
+ int indexes_completed = 0;
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
@@ -2664,15 +2704,32 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ /*
+ * Outsource everything to parallel variant
+ *
+ * See the lazy_vacuum_all_indexes comments
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
vacrel->num_index_scans,
(vacrel->scanned_pages < vacrel->rel_pages));
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
}
+
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
}
/*
@@ -3453,3 +3510,92 @@ restore_vacuum_error_info(LVRelState *vacrel,
vacrel->offnum = saved_vacrel->offnum;
vacrel->phase = saved_vacrel->phase;
}
+
+/*
+ * vacuum_worker_update --- sets the number of indexes processed so far
+ * in a parallel vacuum.
+ */
+void
+vacuum_worker_update(int leader_pid)
+{
+ VacProgressEntry *entry;
+ bool found;
+
+ LWLockAcquire(VacuumWorkerProgressLock, LW_EXCLUSIVE);
+
+ entry = (VacProgressEntry *) hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_ENTER_NULL, &found);
+
+ if (!entry)
+ elog(ERROR, "cannot allocate shared memory for vacuum worker progress");
+
+ if (!found)
+ entry->indexes_processed = 1;
+ else
+ entry->indexes_processed++;
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_end --- remove the leader_pid of a completed parallel vacuum
+ */
+void
+vacuum_worker_end(int leader_pid)
+{
+ LWLockAcquire(VacuumWorkerProgressLock, LW_EXCLUSIVE);
+
+ hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_REMOVE, NULL);
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_end wrapped as an on_shmem_exit callback function
+ */
+void
+vacuum_worker_end_callback(int code, Datum arg)
+{
+ vacuum_worker_end(DatumGetInt32(arg));
+}
+
+/*
+ * set_vaccum_worker_progress --- updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+set_vaccum_worker_progress(Datum *values)
+{
+ VacProgressEntry *entry;
+ int leader_pid = values[0];
+
+ LWLockAcquire(VacuumWorkerProgressLock, LW_SHARED);
+
+ entry = (VacProgressEntry *) hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_FIND, NULL);
+
+ if (entry != NULL)
+ values[PGSTAT_NUM_PROGRESS_COMMON + PROGRESS_VACUUM_INDEXES_COMPLETED] = entry->indexes_processed;
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_init --- initialize this module's shared memory hash
+ * to track the progress of a vacuum worker
+ */
+void
+vacuum_worker_init(void)
+{
+ HASHCTL info;
+ long max_table_size = GetMaxBackends();
+
+ VacuumWorkerProgressHash = NULL;
+
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(VacProgressEntry);
+
+ VacuumWorkerProgressHash = ShmemInitHash("Vacuum Progress Hash",
+ max_table_size,
+ max_table_size,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+}
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 40b7bca5a9..04ce4a45d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1126,7 +1126,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..9b465e12cc 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -844,9 +849,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index cd4ebe2fc5..83ecad4e4a 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -24,6 +24,7 @@
#include "access/twophase.h"
#include "access/xlogrecovery.h"
#include "commands/async.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -296,6 +297,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ vacuum_worker_init();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 6c7cf6c295..9a5fa0a0e0 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+VacuumWorkerProgressLock 48
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index fd993d0d5f..f90c39d9b9 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,8 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/progress.h"
+#include "commands/vacuum.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -452,11 +454,15 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
/*
* Returns command progress information for the named command.
+ *
+ * A command type can optionally define a callback function
+ * which will derive Datum values rather than use values
+ * directly from the backends progress array.
*/
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + 3
+#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + PGSTAT_NUM_PROGRESS_COMMON
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
char *cmd = text_to_cstring(PG_GETARG_TEXT_PP(0));
@@ -543,15 +549,19 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
values[2] = ObjectIdGetDatum(beentry->st_progress_command_target);
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- values[i + 3] = Int64GetDatum(beentry->st_progress_param[i]);
+ values[i + PGSTAT_NUM_PROGRESS_COMMON] = Int64GetDatum(beentry->st_progress_param[i]);
}
else
{
nulls[2] = true;
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- nulls[i + 3] = true;
+ nulls[i + PGSTAT_NUM_PROGRESS_COMMON] = true;
}
+ /* Call the command specific function to override datum values */
+ if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ set_vaccum_worker_progress(values);
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..c0dd1d7ab5 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
@@ -151,4 +153,7 @@
#define PROGRESS_COPY_TYPE_PIPE 3
#define PROGRESS_COPY_TYPE_CALLBACK 4
+/* Number of common fields at the start of progress views */
+#define PGSTAT_NUM_PROGRESS_COMMON 3
+
#endif
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f2..8661516ae2 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -336,4 +336,11 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+/* in commands/vacuumparallel.c */
+extern void vacuum_worker_init(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+extern void set_vaccum_worker_progress(Datum *values);
+
#endif /* VACUUM_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac468568a1..d70a176514 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_replication| SELECT s.pid,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-08 06:04 ` Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-03-08 06:04 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Thu, Mar 3, 2022 at 2:08 PM Imseih (AWS), Sami <[email protected]> wrote:
>
> > >> If the failsafe kicks in midway through a vacuum, the number indexes_total will not be reset to 0. If INDEX_CLEANUP is turned off, then the value will be 0 at the start of the vacuum.
> > >
> > > The way that this works with num_index_scans is that we "round up"
> > > when there has been non-zero work in lazy_vacuum_all_indexes(), but
> > > not if the precheck in lazy_vacuum_all_indexes() fails. That seems
> > > like a good model to generalize from here. Note that this makes
> > > INDEX_CLEANUP=off affect num_index_scans in much the same way as a
> > > VACUUM where the failsafe kicks in very early, during the initial heap
> > > pass. That is, if the failsafe kicks in before we reach lazy_vacuum()
> > > for the first time (which is not unlikely), or even in the
> > > lazy_vacuum_all_indexes() precheck, then num_index_scans will remain
> > > at 0, just like INDEX_CLEANUP=off.
> > >
> > > The actual failsafe WARNING shows num_index_scans, possibly before it
> > > gets incremented one last time (by "rounding up"). So it's reasonably
> > > clear how this all works from that context (assuming that the
> > > autovacuum logging stuff, which reports num_index_scans, outputs a
> > > report for a table where the failsafe kicked in).
>
> > I am confused. If failsafe kicks in during the middle of a vacuum, I
> > (perhaps naively) would expect indexes_total and indexes_processed to go to
> > zero, and I'd expect to no longer see the "vacuuming indexes" and "cleaning
> > up indexes" phases. Otherwise, how would I know that we are now skipping
> > indexes? Of course, you won't have any historical context about the index
> > work done before failsafe kicked in, but IMO it is misleading to still
> > include it in the progress view.
>
> After speaking with Nathan offline, the best forward is to reset indexes_total and indexes_processed to 0 after the start of "vacuuming indexes" or "cleaning up indexes" phase.
+1
+/*
+ * vacuum_worker_init --- initialize this module's shared memory hash
+ * to track the progress of a vacuum worker
+ */
+void
+vacuum_worker_init(void)
+{
+ HASHCTL info;
+ long max_table_size = GetMaxBackends();
+
+ VacuumWorkerProgressHash = NULL;
+
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(VacProgressEntry);
+
+ VacuumWorkerProgressHash = ShmemInitHash("Vacuum Progress Hash",
+
max_table_size,
+
max_table_size,
+
&info,
+
HASH_ELEM | HASH_BLOBS);
+}
It seems to me that creating a shmem hash with max_table_size entries
for parallel vacuum process tracking is too much. IIRC an old patch
had parallel vacuum workers advertise its progress and changed the
pg_stat_progress_vacuum view so that it aggregates the results
including workers' stats. I think it’s better than the current one.
Why did you change that?
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-03-08 15:41 ` Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-08 15:41 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
+ +/*
+ + * vacuum_worker_init --- initialize this module's shared memory hash
+ + * to track the progress of a vacuum worker
+ + */
+ +void
+ +vacuum_worker_init(void)
+ +{
+ + HASHCTL info;
+ + long max_table_size = GetMaxBackends();
+ +
+ + VacuumWorkerProgressHash = NULL;
+ +
+ + info.keysize = sizeof(pid_t);
+ + info.entrysize = sizeof(VacProgressEntry);
+ +
+ + VacuumWorkerProgressHash = ShmemInitHash("Vacuum Progress Hash",
+ +
+ max_table_size,
+ +
+ max_table_size,
+ +
+ &info,
+ +
+ HASH_ELEM | HASH_BLOBS);
+ +}
+ It seems to me that creating a shmem hash with max_table_size entries
+ for parallel vacuum process tracking is too much. IIRC an old patch
+ had parallel vacuum workers advertise its progress and changed the
+ pg_stat_progress_vacuum view so that it aggregates the results
+ including workers' stats. I think it’s better than the current one.
+ Why did you change that?
+ Regards,
I was trying to avoid a shared memory to track completed indexes, but aggregating stats does not work with parallel vacuums. This is because a parallel worker will exit before the vacuum completes causing the aggregated total to be wrong.
For example
Leader_pid advertises it completed 2 indexes
Parallel worker advertises it completed 2 indexes
When aggregating we see 4 indexes completed.
After the parallel worker exits, the aggregation will show only 2 indexes completed.
--
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-09 01:53 ` Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-03-09 01:53 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Wed, Mar 9, 2022 at 12:41 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> + +/*
> + + * vacuum_worker_init --- initialize this module's shared memory hash
> + + * to track the progress of a vacuum worker
> + + */
> + +void
> + +vacuum_worker_init(void)
> + +{
> + + HASHCTL info;
> + + long max_table_size = GetMaxBackends();
> + +
> + + VacuumWorkerProgressHash = NULL;
> + +
> + + info.keysize = sizeof(pid_t);
> + + info.entrysize = sizeof(VacProgressEntry);
> + +
> + + VacuumWorkerProgressHash = ShmemInitHash("Vacuum Progress Hash",
> + +
> + max_table_size,
> + +
> + max_table_size,
> + +
> + &info,
> + +
> + HASH_ELEM | HASH_BLOBS);
> + +}
>
> + It seems to me that creating a shmem hash with max_table_size entries
> + for parallel vacuum process tracking is too much. IIRC an old patch
> + had parallel vacuum workers advertise its progress and changed the
> + pg_stat_progress_vacuum view so that it aggregates the results
> + including workers' stats. I think it’s better than the current one.
> + Why did you change that?
>
> + Regards,
>
> I was trying to avoid a shared memory to track completed indexes, but aggregating stats does not work with parallel vacuums. This is because a parallel worker will exit before the vacuum completes causing the aggregated total to be wrong.
>
> For example
>
> Leader_pid advertises it completed 2 indexes
> Parallel worker advertises it completed 2 indexes
>
> When aggregating we see 4 indexes completed.
>
> After the parallel worker exits, the aggregation will show only 2 indexes completed.
Indeed.
It might have already been discussed but other than using a new shmem
hash for parallel vacuum, I wonder if we can allow workers to change
the leader’s progress information. It would break the assumption that
the backend status entry is modified by its own backend, though. But
it might help for progress updates of other parallel operations too.
This essentially does the same thing as what the current patch does
but it doesn't require a new shmem hash.
Another idea I come up with is that the parallel vacuum leader checks
PVIndStats.status and updates how many indexes are processed to its
progress information. The leader can check it and update the progress
information before and after index vacuuming. And possibly we can add
a callback to the main loop of index AM's bulkdelete and vacuumcleanup
so that the leader can periodically make it up-to-date.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-03-09 02:35 ` Imseih (AWS), Sami <[email protected]>
2022-03-09 02:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-09 02:35 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> Indeed.
> It might have already been discussed but other than using a new shmem
> hash for parallel vacuum, I wonder if we can allow workers to change
> the leader’s progress information. It would break the assumption that
> the backend status entry is modified by its own backend, though. But
> it might help for progress updates of other parallel operations too.
> This essentially does the same thing as what the current patch does
> but it doesn't require a new shmem hash.
I experimented with this idea, but it did not work. The idea would have been to create a pgstat_progress_update function that takes the leader pid, however infrastructure does not exist to allow one backend to manipulate another backends backend status array.
pgstat_fetch_stat_beentry returns a local copy only.
> Another idea I come up with is that the parallel vacuum leader checks
> PVIndStats.status and updates how many indexes are processed to its
> progress information. The leader can check it and update the progress
> information before and after index vacuuming. And possibly we can add
> a callback to the main loop of index AM's bulkdelete and vacuumcleanup
> so that the leader can periodically make it up-to-date.
> Regards,
The PVIndStats idea is also one I experimented with but it did not work. The reason being the backend checking the progress needs to do a shm_toc_lookup to access the data, but they are not prepared to do so.
I have not considered the callback in the index AM's bulkdelete and vacuumcleanup, but I can imagine this is not possible since a leader could be busy vacuuming rather than updating counters, but I may be misunderstanding the suggestion.
--
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-09 02:57 ` Imseih (AWS), Sami <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-09 02:57 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; Justin Pryzby <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; pgsql-hackers
Attached is the latest revision of the patch(s). Renamed the patches correctly for Cfbot.
--
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v3-0001-Show-progress-for-index-vacuums.patch (17.6K, ../../[email protected]/2-v3-0001-Show-progress-for-index-vacuums.patch)
download | inline diff:
From fe37433fd4304ddf19be1f8c2f637f79d7fc2ebb Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Wed, 9 Mar 2022 01:54:08 +0000
Subject: [PATCH v3 1/3] Show progress for index vacuums
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 24 ++++
src/backend/access/heap/vacuumlazy.c | 156 ++++++++++++++++++++++-
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 7 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/utils/adt/pgstatfuncs.c | 16 ++-
src/include/commands/progress.h | 5 +
src/include/commands/vacuum.h | 7 +
src/test/regress/expected/rules.out | 4 +-
10 files changed, 215 insertions(+), 10 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 9fb62fec8e..140fbd5fb9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6227,6 +6227,30 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes already processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 40101e0cb8..4c91e673d1 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -244,6 +245,16 @@ typedef struct LVSavedErrInfo
VacErrPhase phase;
} LVSavedErrInfo;
+/*
+ * Structs for tracking shared Progress information
+ * amongst worker ( and leader ) processes of a vacuum.
+ */
+typedef struct VacProgressEntry
+{
+ pid_t leader_pid;
+ int indexes_total;
+ int indexes_processed;
+} VacProgressEntry;
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
@@ -290,6 +301,7 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
+static HTAB *VacuumWorkerProgressHash;
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
@@ -2305,9 +2317,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/* Report that we are now vacuuming indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
+ int indexes_completed = 1;
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
@@ -2317,6 +2333,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2327,9 +2345,22 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
- vacrel->num_index_scans);
+ /*
+ * Outsource everything to parallel variant.
+ *
+ * * parallel_vacuum_bulkdel_all_indexes will call vacuum_worker_update
+ * which updates shared memory for the index progress. To ensure shared
+ * memory cleanup, do the work with PG_ENSURE_ERROR_CLEANUP.
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs,
+ vacrel->old_live_tuples,
+ vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2339,6 +2370,10 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
allindexes = false;
}
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
/*
* We delete all LP_DEAD items from the first heap pass in all indexes on
* each call here (except calls where we choose to do the failsafe). This
@@ -2617,6 +2652,8 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2649,12 +2686,15 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
/* Report that we are now cleaning up indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
+ /* Advertise the number of indexes we are cleaning up */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
double reltuples = vacrel->new_rel_tuples;
bool estimated_count =
vacrel->scanned_pages < vacrel->rel_pages;
+ int indexes_completed = 1;
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
@@ -2664,15 +2704,32 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ /*
+ * Outsource everything to parallel variant
+ *
+ * See the lazy_vacuum_all_indexes comments
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
vacrel->num_index_scans,
(vacrel->scanned_pages < vacrel->rel_pages));
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
}
+
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
}
/*
@@ -3453,3 +3510,92 @@ restore_vacuum_error_info(LVRelState *vacrel,
vacrel->offnum = saved_vacrel->offnum;
vacrel->phase = saved_vacrel->phase;
}
+
+/*
+ * vacuum_worker_update --- sets the number of indexes processed so far
+ * in a parallel vacuum.
+ */
+void
+vacuum_worker_update(int leader_pid)
+{
+ VacProgressEntry *entry;
+ bool found;
+
+ LWLockAcquire(VacuumWorkerProgressLock, LW_EXCLUSIVE);
+
+ entry = (VacProgressEntry *) hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_ENTER_NULL, &found);
+
+ if (!entry)
+ elog(ERROR, "cannot allocate shared memory for vacuum worker progress");
+
+ if (!found)
+ entry->indexes_processed = 1;
+ else
+ entry->indexes_processed++;
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_end --- remove the leader_pid of a completed parallel vacuum
+ */
+void
+vacuum_worker_end(int leader_pid)
+{
+ LWLockAcquire(VacuumWorkerProgressLock, LW_EXCLUSIVE);
+
+ hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_REMOVE, NULL);
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_end wrapped as an on_shmem_exit callback function
+ */
+void
+vacuum_worker_end_callback(int code, Datum arg)
+{
+ vacuum_worker_end(DatumGetInt32(arg));
+}
+
+/*
+ * set_vaccum_worker_progress --- updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+set_vaccum_worker_progress(Datum *values)
+{
+ VacProgressEntry *entry;
+ int leader_pid = values[0];
+
+ LWLockAcquire(VacuumWorkerProgressLock, LW_SHARED);
+
+ entry = (VacProgressEntry *) hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_FIND, NULL);
+
+ if (entry != NULL)
+ values[PGSTAT_NUM_PROGRESS_COMMON + PROGRESS_VACUUM_INDEXES_COMPLETED] = entry->indexes_processed;
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_init --- initialize this module's shared memory hash
+ * to track the progress of a vacuum worker
+ */
+void
+vacuum_worker_init(void)
+{
+ HASHCTL info;
+ long max_table_size = GetMaxBackends();
+
+ VacuumWorkerProgressHash = NULL;
+
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(VacProgressEntry);
+
+ VacuumWorkerProgressHash = ShmemInitHash("Vacuum Progress Hash",
+ max_table_size,
+ max_table_size,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+}
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 40b7bca5a9..04ce4a45d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1126,7 +1126,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..9b465e12cc 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -844,9 +849,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index cd4ebe2fc5..83ecad4e4a 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -24,6 +24,7 @@
#include "access/twophase.h"
#include "access/xlogrecovery.h"
#include "commands/async.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -296,6 +297,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ vacuum_worker_init();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 6c7cf6c295..9a5fa0a0e0 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+VacuumWorkerProgressLock 48
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index eff45b16f2..ad7327fbd9 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,8 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/progress.h"
+#include "commands/vacuum.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -452,11 +454,15 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
/*
* Returns command progress information for the named command.
+ *
+ * A command type can optionally define a callback function
+ * which will derive Datum values rather than use values
+ * directly from the backends progress array.
*/
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + 3
+#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + PGSTAT_NUM_PROGRESS_COMMON
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
char *cmd = text_to_cstring(PG_GETARG_TEXT_PP(0));
@@ -518,15 +524,19 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
values[2] = ObjectIdGetDatum(beentry->st_progress_command_target);
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- values[i + 3] = Int64GetDatum(beentry->st_progress_param[i]);
+ values[i + PGSTAT_NUM_PROGRESS_COMMON] = Int64GetDatum(beentry->st_progress_param[i]);
}
else
{
nulls[2] = true;
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- nulls[i + 3] = true;
+ nulls[i + PGSTAT_NUM_PROGRESS_COMMON] = true;
}
+ /* Call the command specific function to override datum values */
+ if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ set_vaccum_worker_progress(values);
+
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..c0dd1d7ab5 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
@@ -151,4 +153,7 @@
#define PROGRESS_COPY_TYPE_PIPE 3
#define PROGRESS_COPY_TYPE_CALLBACK 4
+/* Number of common fields at the start of progress views */
+#define PGSTAT_NUM_PROGRESS_COMMON 3
+
#endif
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f2..8661516ae2 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -336,4 +336,11 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+/* in commands/vacuumparallel.c */
+extern void vacuum_worker_init(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+extern void set_vaccum_worker_progress(Datum *values);
+
#endif /* VACUUM_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac468568a1..d70a176514 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_replication| SELECT s.pid,
--
2.32.0
[application/octet-stream] v3-0002-Expose-indexes-being-processed-in-a-VACUUM-operat.patch (20.5K, ../../[email protected]/3-v3-0002-Expose-indexes-being-processed-in-a-VACUUM-operat.patch)
download | inline diff:
From bb6561bf2ba6ba71e4409e2ad9daa034760c068c Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Wed, 9 Mar 2022 01:58:18 +0000
Subject: [PATCH v3 2/3] Expose indexes being processed in a VACUUM operation.
A new view called pg_stat_progress_vacuum_index to show the indexrelid
being vacuumed or cleaned during a vacuum. The view also shows the
number of tuples removed for the index during the vacuuming indexes phase.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
---
doc/src/sgml/monitoring.sgml | 108 ++++++++++++++++++++++++++
src/backend/access/gin/ginvacuum.c | 3 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 15 +++-
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 39 ++++++++++
src/backend/commands/vacuumparallel.c | 12 +++
src/backend/utils/adt/pgstatfuncs.c | 2 +
src/include/commands/progress.h | 17 ++--
src/include/utils/backend_progress.h | 1 +
src/test/regress/expected/rules.out | 34 ++++++++
13 files changed, 233 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 140fbd5fb9..3fdfc6a4ce 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -379,6 +379,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_progress_vacuum_index</structname><indexterm><primary>pg_stat_progress_vacuum_index</primary></indexterm></entry>
+ <entry>One row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum, showing current progress.
+ See <xref linkend='vacuum-progress-reporting'/>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_cluster</structname><indexterm><primary>pg_stat_progress_cluster</primary></indexterm></entry>
<entry>One row for each backend running
@@ -6255,6 +6264,105 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</tgroup>
</table>
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
<table id="vacuum-phases">
<title>VACUUM Phases</title>
<tgroup cols="2">
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..1d5d003780 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,7 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..23dacee52e 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -632,6 +632,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 4c91e673d1..53a9c7f4d6 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -360,6 +360,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
/*
* Get OldestXmin cutoff, which is used to determine which deleted tuples
@@ -2329,12 +2330,19 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being vacuumed in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+ /* Advertise we are done vacuuming indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2348,7 +2356,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* Outsource everything to parallel variant.
*
- * * parallel_vacuum_bulkdel_all_indexes will call vacuum_worker_update
+ * parallel_vacuum_bulkdel_all_indexes will call vacuum_worker_update
* which updates shared memory for the index progress. To ensure shared
* memory cleanup, do the work with PG_ENSURE_ERROR_CLEANUP.
*/
@@ -2701,12 +2709,17 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being cleaned in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+ /* Advertise we are done cleaning indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
}
}
else
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c9b4964c1e..09edf49082 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1273,6 +1273,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 04ce4a45d1..5bf8380ea4 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1131,6 +1131,45 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid,
+ S.datid,
+ S.datname,
+ S.indexrelid,
+ S.leader_pid,
+ CASE S.phase WHEN 2 THEN 'vacuuming indexes'
+ WHEN 4 THEN 'cleaning up indexes'
+ WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
+ END AS phase,
+ S.tuples_removed
+ FROM (
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 IN (2, 4, 7, 8) AND S.param10 > 0
+ UNION ALL
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM_PARALLEL') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ ) AS S
+ WHERE S.phase IN (2, 4, 7, 8) AND S.indexrelid > 0;
+
CREATE VIEW pg_stat_progress_cluster AS
SELECT
S.pid AS pid,
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 9b465e12cc..711fede2d9 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -845,14 +845,19 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ /* Advertise the index we are cleaning or vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL);
vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL);
vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
@@ -888,6 +893,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
*/
indstats->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+ /* Advertise we are no longer vacuuming/cleaning an index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
@@ -972,6 +981,8 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL, RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -1042,6 +1053,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
+ pgstat_progress_end_command();
FreeAccessStrategy(pvs.bstrategy);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index ad7327fbd9..e0e12570f5 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -472,6 +472,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
cmdtype = PROGRESS_COMMAND_VACUUM;
+ else if (pg_strcasecmp(cmd, "VACUUM_PARALLEL") == 0)
+ cmdtype = PROGRESS_COMMAND_VACUUM_PARALLEL;
else if (pg_strcasecmp(cmd, "ANALYZE") == 0)
cmdtype = PROGRESS_COMMAND_ANALYZE;
else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index c0dd1d7ab5..bcbf3279bd 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -27,14 +27,19 @@
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
#define PROGRESS_VACUUM_TOTAL_INDEXES 7
#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
+#define PROGRESS_VACUUM_INDEXRELID 9
+#define PROGRESS_VACUUM_TUPLES_REMOVED 10
+#define PROGRESS_VACUUM_LEADER_PID 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
-#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
-#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
-#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
-#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
-#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
-#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
+#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
+#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
+#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL 7
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL 8
/* Progress parameters for analyze */
#define PROGRESS_ANALYZE_PHASE 0
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..4651e45c40 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -23,6 +23,7 @@ typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_VACUUM_PARALLEL,
PROGRESS_COMMAND_ANALYZE,
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d70a176514..709c7d9613 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2007,6 +2007,40 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ s.datname,
+ s.indexrelid,
+ s.leader_pid,
+ CASE s.phase
+ WHEN 2 THEN 'vacuuming indexes'::text
+ WHEN 4 THEN 'cleaning up indexes'::text
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
+ ELSE NULL::text
+ END AS phase,
+ s.tuples_removed
+ FROM ( SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))
+ WHERE ((s_1.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s_1.param10 > 0))
+ UNION ALL
+ SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM_PARALLEL'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))) s
+ WHERE ((s.phase = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s.indexrelid > 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
[application/octet-stream] v3-0003-Rename-index_vacuum_count-in-pg_stat_pogress_vacu.patch (2.8K, ../../[email protected]/4-v3-0003-Rename-index_vacuum_count-in-pg_stat_pogress_vacu.patch)
download | inline diff:
From 50be6ff7d719046e6fb855934e0b0120ede82627 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Wed, 9 Mar 2022 01:59:15 +0000
Subject: [PATCH v3 3/3] Rename "index_vacuum_count" in pg_stat_pogress_vacuum.
Commit 76d8a1bfd5207d28a4e9fe98a0e1ea7c096d70aa introduces 2 new columns
to track index vacuum/cleanup progress. The columns have "index" in the
name. To make it clear what the existing "index_vacuum_count" column
refers to, which is the cycle count of the index vacuum, this change
renames the column to "index_vacuum_count"
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/catalog/system_views.sql | 2 +-
src/test/regress/expected/rules.out | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 3fdfc6a4ce..263d155044 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6210,7 +6210,7 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>index_vacuum_count</structfield> <type>bigint</type>
+ <structfield>index_vacuum_cycle_count</structfield> <type>bigint</type>
</para>
<para>
Number of completed index vacuum cycles.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5bf8380ea4..1bddc9867e 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 6 THEN 'performing final cleanup'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
- S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+ S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_cycle_count,
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 709c7d9613..bc3bac25b2 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2000,7 +2000,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param2 AS heap_blks_total,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
- s.param5 AS index_vacuum_count,
+ s.param5 AS index_vacuum_cycle_count,
s.param6 AS max_dead_tuples,
s.param7 AS num_dead_tuples,
s.param8 AS indexes_total,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-09 03:15 ` Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-03-09 03:15 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Wed, Mar 9, 2022 at 11:35 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> > Indeed.
>
> > It might have already been discussed but other than using a new shmem
> > hash for parallel vacuum, I wonder if we can allow workers to change
> > the leader’s progress information. It would break the assumption that
> > the backend status entry is modified by its own backend, though. But
> > it might help for progress updates of other parallel operations too.
> > This essentially does the same thing as what the current patch does
> > but it doesn't require a new shmem hash.
>
> I experimented with this idea, but it did not work. The idea would have been to create a pgstat_progress_update function that takes the leader pid, however infrastructure does not exist to allow one backend to manipulate another backends backend status array.
> pgstat_fetch_stat_beentry returns a local copy only.
I think if it's a better approach we can do that including adding a
new infrastructure for it.
>
> > Another idea I come up with is that the parallel vacuum leader checks
> > PVIndStats.status and updates how many indexes are processed to its
> > progress information. The leader can check it and update the progress
> > information before and after index vacuuming. And possibly we can add
> > a callback to the main loop of index AM's bulkdelete and vacuumcleanup
> > so that the leader can periodically make it up-to-date.
>
> > Regards,
>
> The PVIndStats idea is also one I experimented with but it did not work. The reason being the backend checking the progress needs to do a shm_toc_lookup to access the data, but they are not prepared to do so.
What I imagined is that the leader checks how many PVIndStats.status
is PARALLEL_INDVAC_STATUS_COMPLETED and updates the result to its
progress information as indexes_processed. That way, the backend
checking the progress can see it.
>
> I have not considered the callback in the index AM's bulkdelete and vacuumcleanup, but I can imagine this is not possible since a leader could be busy vacuuming rather than updating counters, but I may be misunderstanding the suggestion.
Checking PVIndStats.status values is cheap. Probably the leader can
check it every 1GB index block, for example.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-03-09 22:52 ` Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-09 22:52 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> I think if it's a better approach we can do that including adding a
> new infrastructure for it.
+1 This is a beneficial idea, especially to other progress reporting, but I see this as a separate thread targeting the next major version.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-10 00:52 ` Nathan Bossart <[email protected]>
2022-03-10 01:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Nathan Bossart @ 2022-03-10 00:52 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
I took a look at the latest patch set.
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
Could we avoid resetting it to 0 unless INDEX_CLEANUP was turned off or
failsafe kicked in? It might be nice to know how many indexes the vacuum
intends to process. I don't feel too strongly about this, so if it would
add a lot of complexity, it might be okay as is.
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ vacuum_worker_init();
Don't we also need to add the size of the hash table to
CalculateShmemSize()?
+ * A command type can optionally define a callback function
+ * which will derive Datum values rather than use values
+ * directly from the backends progress array.
I think this can be removed.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
@ 2022-03-10 01:22 ` Imseih (AWS), Sami <[email protected]>
2022-03-10 01:37 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-10 01:22 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> I took a look at the latest patch set.
> + <entry role="catalog_table_entry"><para role="column_definition">
> + <structfield>indexes_total</structfield> <type>bigint</type>
> + </para>
> + <para>
> + The number of indexes to be processed in the
> + <literal>vacuuming indexes</literal>
> + or <literal>cleaning up indexes</literal> phase. It is set to
> + <literal>0</literal> when vacuum is not in any of these phases.
> + </para></entry>
> Could we avoid resetting it to 0 unless INDEX_CLEANUP was turned off or
> failsafe kicked in? It might be nice to know how many indexes the vacuum
> intends to process. I don't feel too strongly about this, so if it would
> add a lot of complexity, it might be okay as is.
Your suggestion is valid. On INDEX_CLEANUP it is set to 0 from the start and when failsafe kicks in it will be reset to 0. I Will remove the reset call for the common index vacuum path.
> BTreeShmemInit();
> SyncScanShmemInit();
> AsyncShmemInit();
> + vacuum_worker_init();
> Don't we also need to add the size of the hash table to
> CalculateShmemSize()?
No, ShmemInitHash takes the min and max size of the hash and in turn calls ShmemInitStruct to setup the shared memory.
> + * A command type can optionally define a callback function
> + * which will derive Datum values rather than use values
> + * directly from the backends progress array.
> I think this can be removed.
Good catch.
--
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-10 01:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-10 01:37 ` Imseih (AWS), Sami <[email protected]>
2022-03-10 21:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-10 01:37 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> > BTreeShmemInit();
> > SyncScanShmemInit();
> > AsyncShmemInit();
> > + vacuum_worker_init();
> > Don't we also need to add the size of the hash table to
> > CalculateShmemSize()?
> No, ShmemInitHash takes the min and max size of the hash and in turn calls ShmemInitStruct to setup the shared memory.
Sorry, I am wrong here. The size needs to be accounted for at startup.
--
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-10 01:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 01:37 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-10 21:30 ` Imseih (AWS), Sami <[email protected]>
2022-03-10 22:36 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-10 21:30 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Attached v4 which includes accounting for the hash size on startup, removal of the no longer needed comment in pgstatfuncs.c and a change in both code/docs to only reset the indexes_total to 0 when failsafe is triggered.
--
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v4-0001-Show-progress-for-index-vacuums.patch (18.1K, ../../[email protected]/2-v4-0001-Show-progress-for-index-vacuums.patch)
download | inline diff:
From 41ab6f0c1d6e43c9373527093424365a274ea020 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 10 Mar 2022 02:58:56 +0000
Subject: [PATCH v4 1/3] Show progress for index vacuums
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 25 ++++
src/backend/access/heap/vacuumlazy.c | 170 ++++++++++++++++++++++-
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 7 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/utils/adt/pgstatfuncs.c | 12 +-
src/include/commands/progress.h | 5 +
src/include/commands/vacuum.h | 8 ++
src/test/regress/expected/rules.out | 4 +-
10 files changed, 228 insertions(+), 10 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 9fb62fec8e..1acc741da9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6227,6 +6227,31 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when there are no indexes to process
+ or when failsafe is triggered during the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes already processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 40101e0cb8..ccdd41ff55 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -244,6 +245,16 @@ typedef struct LVSavedErrInfo
VacErrPhase phase;
} LVSavedErrInfo;
+/*
+ * Structs for tracking shared Progress information
+ * amongst worker ( and leader ) processes of a vacuum.
+ */
+typedef struct VacProgressEntry
+{
+ pid_t leader_pid;
+ int indexes_total;
+ int indexes_processed;
+} VacProgressEntry;
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
@@ -290,6 +301,7 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
+static HTAB *VacuumWorkerProgressHash;
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
@@ -2305,9 +2317,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/* Report that we are now vacuuming indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
+ int indexes_completed = 1;
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
@@ -2317,6 +2333,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2327,9 +2345,22 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
- vacrel->num_index_scans);
+ /*
+ * Outsource everything to parallel variant.
+ *
+ * parallel_vacuum_bulkdel_all_indexes will call vacuum_worker_update
+ * which updates shared memory for the index progress. To ensure shared
+ * memory cleanup, do the work with PG_ENSURE_ERROR_CLEANUP.
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs,
+ vacrel->old_live_tuples,
+ vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2339,6 +2370,9 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
allindexes = false;
}
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
/*
* We delete all LP_DEAD items from the first heap pass in all indexes on
* each call here (except calls where we choose to do the failsafe). This
@@ -2617,6 +2651,8 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2649,12 +2685,15 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
/* Report that we are now cleaning up indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
+ /* Advertise the number of indexes we are cleaning up */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
double reltuples = vacrel->new_rel_tuples;
bool estimated_count =
vacrel->scanned_pages < vacrel->rel_pages;
+ int indexes_completed = 1;
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
@@ -2664,15 +2703,29 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ /*
+ * Outsource everything to parallel variant
+ *
+ * See the lazy_vacuum_all_indexes comments
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
vacrel->num_index_scans,
(vacrel->scanned_pages < vacrel->rel_pages));
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
}
+
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
}
/*
@@ -3453,3 +3506,110 @@ restore_vacuum_error_info(LVRelState *vacrel,
vacrel->offnum = saved_vacrel->offnum;
vacrel->phase = saved_vacrel->phase;
}
+
+/*
+ * vacuum_worker_update --- sets the number of indexes processed so far
+ * in a parallel vacuum.
+ */
+void
+vacuum_worker_update(int leader_pid)
+{
+ VacProgressEntry *entry;
+ bool found;
+
+ LWLockAcquire(VacuumWorkerProgressLock, LW_EXCLUSIVE);
+
+ entry = (VacProgressEntry *) hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_ENTER_NULL, &found);
+
+ if (!entry)
+ elog(ERROR, "cannot allocate shared memory for vacuum worker progress");
+
+ if (!found)
+ entry->indexes_processed = 1;
+ else
+ entry->indexes_processed++;
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_end --- remove the leader_pid of a completed parallel vacuum
+ */
+void
+vacuum_worker_end(int leader_pid)
+{
+ LWLockAcquire(VacuumWorkerProgressLock, LW_EXCLUSIVE);
+
+ hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_REMOVE, NULL);
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_end wrapped as an on_shmem_exit callback function
+ */
+void
+vacuum_worker_end_callback(int code, Datum arg)
+{
+ vacuum_worker_end(DatumGetInt32(arg));
+}
+
+/*
+ * set_vaccum_worker_progress --- updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+set_vaccum_worker_progress(Datum *values)
+{
+ VacProgressEntry *entry;
+ int leader_pid = values[0];
+
+ LWLockAcquire(VacuumWorkerProgressLock, LW_SHARED);
+
+ entry = (VacProgressEntry *) hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_FIND, NULL);
+
+ if (entry != NULL)
+ values[PGSTAT_NUM_PROGRESS_COMMON + PROGRESS_VACUUM_INDEXES_COMPLETED] = entry->indexes_processed;
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * VacuumWorkerProgressShmemInit --- initialize this module's shared memory hash
+ * to track the progress of a vacuum worker
+ */
+void
+VacuumWorkerProgressShmemInit(void)
+{
+ HASHCTL info;
+ long max_table_size = GetMaxBackends();
+
+ VacuumWorkerProgressHash = NULL;
+
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(VacProgressEntry);
+
+ VacuumWorkerProgressHash = ShmemInitHash("Vacuum Progress Hash",
+ max_table_size,
+ max_table_size,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+}
+
+/*
+ * VacuumWorkerProgressShmemSize --- estimate shared-memory space for
+ * the worker progress table
+ */
+Size
+VacuumWorkerProgressShmemSize(void)
+{
+ Size size = 0;
+ long max_table_size;
+
+ /* vacuum worker progress hash table */
+ max_table_size = GetMaxBackends();
+ size = add_size(size, hash_estimate_size(max_table_size,
+ sizeof(VacProgressEntry)));
+
+ return size;
+}
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 40b7bca5a9..04ce4a45d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1126,7 +1126,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..9b465e12cc 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -844,9 +849,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index cd4ebe2fc5..a4bd6a14a3 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -24,6 +24,7 @@
#include "access/twophase.h"
#include "access/xlogrecovery.h"
#include "commands/async.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -145,6 +146,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, VacuumWorkerProgressShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -296,6 +298,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ VacuumWorkerProgressShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 6c7cf6c295..9a5fa0a0e0 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+VacuumWorkerProgressLock 48
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index eff45b16f2..37696fb026 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,8 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/progress.h"
+#include "commands/vacuum.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -456,7 +458,7 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + 3
+#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + PGSTAT_NUM_PROGRESS_COMMON
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
char *cmd = text_to_cstring(PG_GETARG_TEXT_PP(0));
@@ -518,15 +520,19 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
values[2] = ObjectIdGetDatum(beentry->st_progress_command_target);
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- values[i + 3] = Int64GetDatum(beentry->st_progress_param[i]);
+ values[i + PGSTAT_NUM_PROGRESS_COMMON] = Int64GetDatum(beentry->st_progress_param[i]);
}
else
{
nulls[2] = true;
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- nulls[i + 3] = true;
+ nulls[i + PGSTAT_NUM_PROGRESS_COMMON] = true;
}
+ /* Call the command specific function to override datum values */
+ if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ set_vaccum_worker_progress(values);
+
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..c0dd1d7ab5 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
@@ -151,4 +153,7 @@
#define PROGRESS_COPY_TYPE_PIPE 3
#define PROGRESS_COPY_TYPE_CALLBACK 4
+/* Number of common fields at the start of progress views */
+#define PGSTAT_NUM_PROGRESS_COMMON 3
+
#endif
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f2..0ef360a43d 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -336,4 +336,12 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+/* in commands/vacuumparallel.c */
+extern void VacuumWorkerProgressShmemInit(void);
+extern Size VacuumWorkerProgressShmemSize(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+extern void set_vaccum_worker_progress(Datum *values);
+
#endif /* VACUUM_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac468568a1..d70a176514 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_replication| SELECT s.pid,
--
2.32.0
[application/octet-stream] v4-0002-Expose-indexes-being-processed-in-a-VACUUM-operat.patch (20.1K, ../../[email protected]/3-v4-0002-Expose-indexes-being-processed-in-a-VACUUM-operat.patch)
download | inline diff:
From 40a1da09824ba97e6a34ed42f9aab07af5363d5f Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 10 Mar 2022 03:32:32 +0000
Subject: [PATCH v4 2/3] Expose indexes being processed in a VACUUM operation.
A new view called pg_stat_progress_vacuum_index to show the indexrelid
being vacuumed or cleaned during a vacuum. The view also shows the
number of tuples removed for the index during the vacuuming indexes phase.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
---
doc/src/sgml/monitoring.sgml | 108 ++++++++++++++++++++++++++
src/backend/access/gin/ginvacuum.c | 3 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 14 ++++
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 39 ++++++++++
src/backend/commands/vacuumparallel.c | 12 +++
src/backend/utils/adt/pgstatfuncs.c | 2 +
src/include/commands/progress.h | 17 ++--
src/include/utils/backend_progress.h | 1 +
src/test/regress/expected/rules.out | 34 ++++++++
13 files changed, 233 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1acc741da9..240d2438d5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -379,6 +379,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_progress_vacuum_index</structname><indexterm><primary>pg_stat_progress_vacuum_index</primary></indexterm></entry>
+ <entry>One row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum, showing current progress.
+ See <xref linkend='vacuum-progress-reporting'/>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_cluster</structname><indexterm><primary>pg_stat_progress_cluster</primary></indexterm></entry>
<entry>One row for each backend running
@@ -6256,6 +6265,105 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</tgroup>
</table>
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
<table id="vacuum-phases">
<title>VACUUM Phases</title>
<tgroup cols="2">
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..1d5d003780 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,7 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..23dacee52e 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -632,6 +632,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index ccdd41ff55..abf9109f7b 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -360,6 +360,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
/*
* Get OldestXmin cutoff, which is used to determine which deleted tuples
@@ -2329,12 +2330,19 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being vacuumed in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+ /* Advertise we are done vacuuming indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2700,11 +2708,17 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being cleaned in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+
+ /* Advertise we are done cleaning indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
}
}
else
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c9b4964c1e..09edf49082 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1273,6 +1273,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 04ce4a45d1..5bf8380ea4 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1131,6 +1131,45 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid,
+ S.datid,
+ S.datname,
+ S.indexrelid,
+ S.leader_pid,
+ CASE S.phase WHEN 2 THEN 'vacuuming indexes'
+ WHEN 4 THEN 'cleaning up indexes'
+ WHEN 7 THEN 'vacuuming indexes'
+ WHEN 8 THEN 'cleaning up indexes'
+ END AS phase,
+ S.tuples_removed
+ FROM (
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 IN (2, 4, 7, 8) AND S.param10 > 0
+ UNION ALL
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ S.param1 AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM_PARALLEL') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ ) AS S
+ WHERE S.phase IN (2, 4, 7, 8) AND S.indexrelid > 0;
+
CREATE VIEW pg_stat_progress_cluster AS
SELECT
S.pid AS pid,
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 9b465e12cc..711fede2d9 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -845,14 +845,19 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ /* Advertise the index we are cleaning or vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL);
vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL);
vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
@@ -888,6 +893,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
*/
indstats->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+ /* Advertise we are no longer vacuuming/cleaning an index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
@@ -972,6 +981,8 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL, RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -1042,6 +1053,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
+ pgstat_progress_end_command();
FreeAccessStrategy(pvs.bstrategy);
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 37696fb026..a1ef9aef49 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -468,6 +468,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
/* Translate command name into command type code. */
if (pg_strcasecmp(cmd, "VACUUM") == 0)
cmdtype = PROGRESS_COMMAND_VACUUM;
+ else if (pg_strcasecmp(cmd, "VACUUM_PARALLEL") == 0)
+ cmdtype = PROGRESS_COMMAND_VACUUM_PARALLEL;
else if (pg_strcasecmp(cmd, "ANALYZE") == 0)
cmdtype = PROGRESS_COMMAND_ANALYZE;
else if (pg_strcasecmp(cmd, "CLUSTER") == 0)
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index c0dd1d7ab5..bcbf3279bd 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -27,14 +27,19 @@
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
#define PROGRESS_VACUUM_TOTAL_INDEXES 7
#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
+#define PROGRESS_VACUUM_INDEXRELID 9
+#define PROGRESS_VACUUM_TUPLES_REMOVED 10
+#define PROGRESS_VACUUM_LEADER_PID 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
-#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
-#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
-#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
-#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
-#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
-#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX 2
+#define PROGRESS_VACUUM_PHASE_VACUUM_HEAP 3
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP 4
+#define PROGRESS_VACUUM_PHASE_TRUNCATE 5
+#define PROGRESS_VACUUM_PHASE_FINAL_CLEANUP 6
+#define PROGRESS_VACUUM_PHASE_VACUUM_INDEX_PARALLEL 7
+#define PROGRESS_VACUUM_PHASE_INDEX_CLEANUP_PARALLEL 8
/* Progress parameters for analyze */
#define PROGRESS_ANALYZE_PHASE 0
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..4651e45c40 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -23,6 +23,7 @@ typedef enum ProgressCommandType
{
PROGRESS_COMMAND_INVALID,
PROGRESS_COMMAND_VACUUM,
+ PROGRESS_COMMAND_VACUUM_PARALLEL,
PROGRESS_COMMAND_ANALYZE,
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d70a176514..709c7d9613 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2007,6 +2007,40 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ s.datname,
+ s.indexrelid,
+ s.leader_pid,
+ CASE s.phase
+ WHEN 2 THEN 'vacuuming indexes'::text
+ WHEN 4 THEN 'cleaning up indexes'::text
+ WHEN 7 THEN 'vacuuming indexes'::text
+ WHEN 8 THEN 'cleaning up indexes'::text
+ ELSE NULL::text
+ END AS phase,
+ s.tuples_removed
+ FROM ( SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))
+ WHERE ((s_1.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s_1.param10 > 0))
+ UNION ALL
+ SELECT s_1.pid,
+ s_1.datid,
+ d.datname,
+ s_1.param10 AS indexrelid,
+ s_1.param12 AS leader_pid,
+ s_1.param1 AS phase,
+ s_1.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM_PARALLEL'::text) s_1(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s_1.datid = d.oid)))) s
+ WHERE ((s.phase = ANY (ARRAY[(2)::bigint, (4)::bigint, (7)::bigint, (8)::bigint])) AND (s.indexrelid > 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
[application/octet-stream] v4-0003-Rename-index_vacuum_count-in-pg_stat_pogress_vacu.patch (2.8K, ../../[email protected]/4-v4-0003-Rename-index_vacuum_count-in-pg_stat_pogress_vacu.patch)
download | inline diff:
From 7aa0140b73b448b64852235e36e34e23b3f6557d Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 10 Mar 2022 03:33:31 +0000
Subject: [PATCH v4 3/3] Rename "index_vacuum_count" in pg_stat_pogress_vacuum.
Commit 76d8a1bfd5207d28a4e9fe98a0e1ea7c096d70aa introduces 2 new columns
to track index vacuum/cleanup progress. The columns have "index" in the
name. To make it clear what the existing "index_vacuum_count" column
refers to, which is the cycle count of the index vacuum, this change
renames the column to "index_vacuum_count"
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/catalog/system_views.sql | 2 +-
src/test/regress/expected/rules.out | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 240d2438d5..1e5af0b400 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6210,7 +6210,7 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>index_vacuum_count</structfield> <type>bigint</type>
+ <structfield>index_vacuum_cycle_count</structfield> <type>bigint</type>
</para>
<para>
Number of completed index vacuum cycles.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5bf8380ea4..1bddc9867e 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 6 THEN 'performing final cleanup'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
- S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+ S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_cycle_count,
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 709c7d9613..bc3bac25b2 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2000,7 +2000,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param2 AS heap_blks_total,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
- s.param5 AS index_vacuum_count,
+ s.param5 AS index_vacuum_cycle_count,
s.param6 AS max_dead_tuples,
s.param7 AS num_dead_tuples,
s.param8 AS indexes_total,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-10 01:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 01:37 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 21:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-10 22:36 ` Nathan Bossart <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Nathan Bossart @ 2022-03-10 22:36 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Thu, Mar 10, 2022 at 09:30:57PM +0000, Imseih (AWS), Sami wrote:
> Attached v4 which includes accounting for the hash size on startup, removal of the no longer needed comment in pgstatfuncs.c and a change in both code/docs to only reset the indexes_total to 0 when failsafe is triggered.
Thanks for the new patch set.
+/*
+ * Structs for tracking shared Progress information
+ * amongst worker ( and leader ) processes of a vacuum.
+ */
nitpick: Can we remove the extra spaces in the parentheses?
+ if (entry != NULL)
+ values[PGSTAT_NUM_PROGRESS_COMMON + PROGRESS_VACUUM_INDEXES_COMPLETED] = entry->indexes_processed;
What does it mean if there isn't an entry in the map? Is this actually
expected, or should we ERROR instead?
+ /* vacuum worker progress hash table */
+ max_table_size = GetMaxBackends();
+ size = add_size(size, hash_estimate_size(max_table_size,
+ sizeof(VacProgressEntry)));
I think the number of entries should be shared between
VacuumWorkerProgressShmemInit() and VacuumWorkerProgressShmemSize().
Otherwise, we might update one and not the other.
+ /* Call the command specific function to override datum values */
+ if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ set_vaccum_worker_progress(values);
I think we should elaborate a bit more in this comment. It's difficult to
follow what this is doing without referencing the comment above
set_vacuum_worker_progress().
IMO the patches are in decent shape, and this should likely be marked as
ready-for-committer in the near future. Before doing so, I think we should
check that Sawada-san is okay with moving the deeper infrastructure changes
to a separate threaḋ.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
@ 2022-03-12 07:00 ` Imseih (AWS), Sami <[email protected]>
2022-03-12 21:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
1 sibling, 2 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-12 07:00 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> nitpick: Can we remove the extra spaces in the parentheses?
fixed
> What does it mean if there isn't an entry in the map? Is this actually
> expected, or should we ERROR instead?
I cleaned up the code here and added comments.
> I think the number of entries should be shared between
> VacuumWorkerProgressShmemInit() and VacuumWorkerProgressShmemSize().
> Otherwise, we might update one and not the other.
Fixed
> I think we should elaborate a bit more in this comment. It's difficult to
> follow what this is doing without referencing the comment above
> set_vacuum_worker_progress().
More comments added
I also simplified the 0002 patch as well.
--
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v5-0001-Show-progress-for-index-vacuums.patch (18.9K, ../../[email protected]/2-v5-0001-Show-progress-for-index-vacuums.patch)
download | inline diff:
From f5b0c2f62e45e78326b038d90d741ee6260e5f9e Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Sat, 12 Mar 2022 01:38:10 +0000
Subject: [PATCH 1/1] Show progress for index vacuums
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 25 +++
src/backend/access/heap/vacuumlazy.c | 186 ++++++++++++++++++++++-
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 7 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/utils/adt/pgstatfuncs.c | 19 ++-
src/include/commands/progress.h | 5 +
src/include/commands/vacuum.h | 8 +
src/test/regress/expected/rules.out | 4 +-
10 files changed, 251 insertions(+), 10 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 9fb62fec8e..1acc741da9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6227,6 +6227,31 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when there are no indexes to process
+ or when failsafe is triggered during the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes already processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 40101e0cb8..3075f36c26 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -126,6 +127,11 @@
*/
#define ParallelVacuumIsActive(vacrel) ((vacrel)->pvs != NULL)
+/*
+ * Set the size of the progress hash table
+ */
+#define VACUUM_WORKER_PROGRESS_NUM_ENTRIES (GetMaxBackends())
+
/* Phases of vacuum during which we report error context. */
typedef enum
{
@@ -244,6 +250,16 @@ typedef struct LVSavedErrInfo
VacErrPhase phase;
} LVSavedErrInfo;
+/*
+ * Structs for tracking shared Progress information
+ * amongst worker (and leader) processes of a vacuum.
+ */
+typedef struct VacProgressEntry
+{
+ pid_t leader_pid;
+ int indexes_total;
+ int indexes_processed;
+} VacProgressEntry;
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
@@ -290,6 +306,7 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
+static HTAB *VacuumWorkerProgressHash;
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
@@ -2305,9 +2322,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/* Report that we are now vacuuming indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
+ int indexes_completed = 1;
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
@@ -2317,6 +2338,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2327,9 +2350,22 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
- vacrel->num_index_scans);
+ /*
+ * Outsource everything to parallel variant.
+ *
+ * parallel_vacuum_bulkdel_all_indexes will call vacuum_worker_update
+ * which updates shared memory for the index progress. To ensure shared
+ * memory cleanup, do the work with PG_ENSURE_ERROR_CLEANUP.
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs,
+ vacrel->old_live_tuples,
+ vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2339,6 +2375,9 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
allindexes = false;
}
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
/*
* We delete all LP_DEAD items from the first heap pass in all indexes on
* each call here (except calls where we choose to do the failsafe). This
@@ -2617,6 +2656,8 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2649,12 +2690,15 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
/* Report that we are now cleaning up indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
+ /* Advertise the number of indexes we are cleaning up */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
double reltuples = vacrel->new_rel_tuples;
bool estimated_count =
vacrel->scanned_pages < vacrel->rel_pages;
+ int indexes_completed = 1;
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
@@ -2664,15 +2708,29 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ /*
+ * Outsource everything to parallel variant
+ *
+ * See the lazy_vacuum_all_indexes comments
+ */
+ PG_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
vacrel->num_index_scans,
(vacrel->scanned_pages < vacrel->rel_pages));
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(vacuum_worker_end_callback, Int32GetDatum(MyProcPid));
+ vacuum_worker_end(MyProcPid);
}
+
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
}
/*
@@ -3453,3 +3511,121 @@ restore_vacuum_error_info(LVRelState *vacrel,
vacrel->offnum = saved_vacrel->offnum;
vacrel->phase = saved_vacrel->phase;
}
+
+/*
+ * vacuum_worker_update --- sets the number of indexes processed so far
+ * in a parallel vacuum.
+ */
+void
+vacuum_worker_update(int leader_pid)
+{
+ VacProgressEntry *entry;
+ bool found;
+
+ LWLockAcquire(VacuumWorkerProgressLock, LW_EXCLUSIVE);
+
+ entry = (VacProgressEntry *) hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_ENTER_NULL, &found);
+
+ if (!entry)
+ elog(ERROR, "cannot allocate shared memory for vacuum worker progress");
+
+ if (!found)
+ entry->indexes_processed = 1;
+ else
+ entry->indexes_processed++;
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_end --- remove the leader_pid of a completed parallel vacuum
+ */
+void
+vacuum_worker_end(int leader_pid)
+{
+ LWLockAcquire(VacuumWorkerProgressLock, LW_EXCLUSIVE);
+
+ /*
+ * Remove from hashtable. It should always be present,
+ * but don't complain if it's not.
+ */
+ hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_REMOVE, NULL);
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * vacuum_worker_end wrapped as an on_shmem_exit callback function
+ */
+void
+vacuum_worker_end_callback(int code, Datum arg)
+{
+ vacuum_worker_end(DatumGetInt32(arg));
+}
+
+/*
+ * set_vaccum_worker_progress --- updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+set_vaccum_worker_progress(Datum *values)
+{
+ VacProgressEntry *entry;
+ int leader_pid = values[0];
+
+ LWLockAcquire(VacuumWorkerProgressLock, LW_SHARED);
+
+ entry = (VacProgressEntry *) hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_FIND, NULL);
+
+ /*
+ * If an entry is not found, it is because we looked at a pid that is not involved in parallel vacuum,
+ * therefore release the read lock and break. For non-parallel vacuums, the indexes_completed is
+ * set in lazy_vacuum_all_indexes.
+ */
+ if (!entry)
+ {
+ LWLockRelease(VacuumWorkerProgressLock);
+ return;
+ }
+
+ values[PGSTAT_NUM_PROGRESS_COMMON + PROGRESS_VACUUM_INDEXES_COMPLETED] = entry->indexes_processed;
+
+ LWLockRelease(VacuumWorkerProgressLock);
+}
+
+/*
+ * VacuumWorkerProgressShmemInit --- initialize this module's shared memory hash
+ * to track the progress of a vacuum worker
+ */
+void
+VacuumWorkerProgressShmemInit(void)
+{
+ HASHCTL info;
+
+ VacuumWorkerProgressHash = NULL;
+
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(VacProgressEntry);
+
+ VacuumWorkerProgressHash = ShmemInitHash("Vacuum Progress Hash",
+ VACUUM_WORKER_PROGRESS_NUM_ENTRIES,
+ VACUUM_WORKER_PROGRESS_NUM_ENTRIES,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+}
+
+/*
+ * VacuumWorkerProgressShmemSize --- estimate shared-memory space for
+ * the worker progress table
+ */
+Size
+VacuumWorkerProgressShmemSize(void)
+{
+ Size size = 0;
+
+ /* vacuum worker progress hash table */
+ size = add_size(size, hash_estimate_size(VACUUM_WORKER_PROGRESS_NUM_ENTRIES,
+ sizeof(VacProgressEntry)));
+
+ return size;
+}
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 40b7bca5a9..04ce4a45d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1126,7 +1126,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..9b465e12cc 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -844,9 +849,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ vacuum_worker_update(pvs->shared->leader_pid);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index cd4ebe2fc5..a4bd6a14a3 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -24,6 +24,7 @@
#include "access/twophase.h"
#include "access/xlogrecovery.h"
#include "commands/async.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -145,6 +146,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, VacuumWorkerProgressShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -296,6 +298,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ VacuumWorkerProgressShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 6c7cf6c295..9a5fa0a0e0 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+VacuumWorkerProgressLock 48
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index eff45b16f2..5526005c47 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,8 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/progress.h"
+#include "commands/vacuum.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -456,7 +458,7 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + 3
+#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + PGSTAT_NUM_PROGRESS_COMMON
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
char *cmd = text_to_cstring(PG_GETARG_TEXT_PP(0));
@@ -518,15 +520,26 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
values[2] = ObjectIdGetDatum(beentry->st_progress_command_target);
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- values[i + 3] = Int64GetDatum(beentry->st_progress_param[i]);
+ values[i + PGSTAT_NUM_PROGRESS_COMMON] = Int64GetDatum(beentry->st_progress_param[i]);
}
else
{
nulls[2] = true;
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- nulls[i + 3] = true;
+ nulls[i + PGSTAT_NUM_PROGRESS_COMMON] = true;
}
+ /*
+ * Before putting values in a tuple store, call a command specific function
+ * to populate datum values that are not set directly in the backend status array.
+ */
+ if (pg_strcasecmp(cmd, "VACUUM") == 0)
+ /*
+ * set_vaccum_worker_progress sets indexes_completed from vacuum progress
+ * shared memory.
+ */
+ set_vaccum_worker_progress(values);
+
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..c0dd1d7ab5 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
@@ -151,4 +153,7 @@
#define PROGRESS_COPY_TYPE_PIPE 3
#define PROGRESS_COPY_TYPE_CALLBACK 4
+/* Number of common fields at the start of progress views */
+#define PGSTAT_NUM_PROGRESS_COMMON 3
+
#endif
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f2..0ef360a43d 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -336,4 +336,12 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+/* in commands/vacuumparallel.c */
+extern void VacuumWorkerProgressShmemInit(void);
+extern Size VacuumWorkerProgressShmemSize(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+extern void set_vaccum_worker_progress(Datum *values);
+
#endif /* VACUUM_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac468568a1..d70a176514 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_replication| SELECT s.pid,
--
2.32.0
[application/octet-stream] v5-0002-Expose-indexes-being-processed-in-a-VACUUM-operat.patch (16.7K, ../../[email protected]/3-v5-0002-Expose-indexes-being-processed-in-a-VACUUM-operat.patch)
download | inline diff:
From e139f9d59dd87ef7253a308ccaae77261494c884 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Sat, 12 Mar 2022 06:23:07 +0000
Subject: [PATCH 1/1] Expose indexes being processed in a VACUUM operation.
A new view called pg_stat_progress_vacuum_index to show the indexrelid
being vacuumed or cleaned during a vacuum. The view also shows the
number of tuples removed for the index during the vacuuming indexes phase.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
---
doc/src/sgml/monitoring.sgml | 108 ++++++++++++++++++++++++++
src/backend/access/gin/ginvacuum.c | 3 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 14 ++++
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 18 ++++-
src/backend/commands/vacuumparallel.c | 12 +++
src/include/commands/progress.h | 3 +
src/test/regress/expected/rules.out | 17 +++-
11 files changed, 182 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1acc741da9..240d2438d5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -379,6 +379,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_progress_vacuum_index</structname><indexterm><primary>pg_stat_progress_vacuum_index</primary></indexterm></entry>
+ <entry>One row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum, showing current progress.
+ See <xref linkend='vacuum-progress-reporting'/>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_cluster</structname><indexterm><primary>pg_stat_progress_cluster</primary></indexterm></entry>
<entry>One row for each backend running
@@ -6256,6 +6265,105 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</tgroup>
</table>
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
<table id="vacuum-phases">
<title>VACUUM Phases</title>
<tgroup cols="2">
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..1d5d003780 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,7 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..23dacee52e 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -632,6 +632,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 3075f36c26..e1dbb36533 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -365,6 +365,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
/*
* Get OldestXmin cutoff, which is used to determine which deleted tuples
@@ -2334,12 +2335,19 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being vacuumed in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+ /* Advertise we are done vacuuming indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2705,11 +2713,17 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being cleaned in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed++);
+
+ /* Advertise we are done cleaning indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
}
}
else
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c9b4964c1e..09edf49082 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1273,6 +1273,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 04ce4a45d1..a9e30b2248 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1129,7 +1129,23 @@ CREATE VIEW pg_stat_progress_vacuum AS
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
- LEFT JOIN pg_database D ON S.datid = D.oid;
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.pid = S.param12;
+
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ CASE S.param1 WHEN 2 THEN 'vacuuming indexes'
+ WHEN 4 THEN 'cleaning up indexes'
+ END AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 IN (2, 4) AND S.param10 > 0;
CREATE VIEW pg_stat_progress_cluster AS
SELECT
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 9b465e12cc..14ffb4df6c 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -845,13 +845,18 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ /* Advertise the index we are cleaning or vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
vacuum_worker_update(pvs->shared->leader_pid);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
vacuum_worker_update(pvs->shared->leader_pid);
break;
@@ -888,6 +893,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
*/
indstats->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+ /* Advertise we are no longer vacuuming/cleaning an index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
@@ -972,6 +981,8 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM, RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -1042,6 +1053,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
+ pgstat_progress_end_command();
FreeAccessStrategy(pvs.bstrategy);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index c0dd1d7ab5..dcbb79072f 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -27,6 +27,9 @@
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
#define PROGRESS_VACUUM_TOTAL_INDEXES 7
#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
+#define PROGRESS_VACUUM_INDEXRELID 9
+#define PROGRESS_VACUUM_TUPLES_REMOVED 10
+#define PROGRESS_VACUUM_LEADER_PID 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d70a176514..d2c66bf034 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2006,7 +2006,22 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param8 AS indexes_total,
s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
- LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE (s.pid = s.param12);
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.param10 AS indexrelid,
+ s.param12 AS leader_pid,
+ CASE s.param1
+ WHEN 2 THEN 'vacuuming indexes'::text
+ WHEN 4 THEN 'cleaning up indexes'::text
+ ELSE NULL::text
+ END AS phase,
+ s.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE ((s.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint])) AND (s.param10 > 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
[application/octet-stream] v5-0003-Rename-index_vacuum_count-in-pg_stat_pogress_vacu.patch (2.8K, ../../[email protected]/4-v5-0003-Rename-index_vacuum_count-in-pg_stat_pogress_vacu.patch)
download | inline diff:
From 7aa0140b73b448b64852235e36e34e23b3f6557d Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 10 Mar 2022 03:33:31 +0000
Subject: [PATCH v4 3/3] Rename "index_vacuum_count" in pg_stat_pogress_vacuum.
Commit 76d8a1bfd5207d28a4e9fe98a0e1ea7c096d70aa introduces 2 new columns
to track index vacuum/cleanup progress. The columns have "index" in the
name. To make it clear what the existing "index_vacuum_count" column
refers to, which is the cycle count of the index vacuum, this change
renames the column to "index_vacuum_count"
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/catalog/system_views.sql | 2 +-
src/test/regress/expected/rules.out | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 240d2438d5..1e5af0b400 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6210,7 +6210,7 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>index_vacuum_count</structfield> <type>bigint</type>
+ <structfield>index_vacuum_cycle_count</structfield> <type>bigint</type>
</para>
<para>
Number of completed index vacuum cycles.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5bf8380ea4..1bddc9867e 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 6 THEN 'performing final cleanup'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
- S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+ S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_cycle_count,
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 709c7d9613..bc3bac25b2 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2000,7 +2000,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param2 AS heap_blks_total,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
- s.param5 AS index_vacuum_count,
+ s.param5 AS index_vacuum_cycle_count,
s.param6 AS max_dead_tuples,
s.param7 AS num_dead_tuples,
s.param8 AS indexes_total,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-12 21:20 ` Nathan Bossart <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Nathan Bossart @ 2022-03-12 21:20 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Sat, Mar 12, 2022 at 07:00:06AM +0000, Imseih (AWS), Sami wrote:
>> nitpick: Can we remove the extra spaces in the parentheses?
>
> fixed
>
>> What does it mean if there isn't an entry in the map? Is this actually
>> expected, or should we ERROR instead?
>
> I cleaned up the code here and added comments.
>
>> I think the number of entries should be shared between
>> VacuumWorkerProgressShmemInit() and VacuumWorkerProgressShmemSize().
>> Otherwise, we might update one and not the other.
>
> Fixed
>
>> I think we should elaborate a bit more in this comment. It's difficult to
>> follow what this is doing without referencing the comment above
>> set_vacuum_worker_progress().
>
> More comments added
>
> I also simplified the 0002 patch as well.
These patches look pretty good to me. Barring additional feedback, I
intend to mark this as ready-for-committer early next week.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-14 01:44 ` Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-03-14 01:44 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Sat, Mar 12, 2022 at 4:00 PM Imseih (AWS), Sami <[email protected]> wrote:
>
> > nitpick: Can we remove the extra spaces in the parentheses?
>
> fixed
>
> > What does it mean if there isn't an entry in the map? Is this actually
> > expected, or should we ERROR instead?
>
> I cleaned up the code here and added comments.
>
> > I think the number of entries should be shared between
> > VacuumWorkerProgressShmemInit() and VacuumWorkerProgressShmemSize().
> > Otherwise, we might update one and not the other.
>
> Fixed
>
> > I think we should elaborate a bit more in this comment. It's difficult to
> > follow what this is doing without referencing the comment above
> > set_vacuum_worker_progress().
>
> More comments added
>
> I also simplified the 0002 patch as well.
I'm still unsure the current design of 0001 patch is better than other
approaches we’ve discussed. Even users who don't use parallel vacuum
are forced to allocate shared memory for index vacuum progress, with
GetMaxBackends() entries from the beginning. Also, it’s likely to
extend the progress tracking feature for other parallel operations in
the future but I think the current design is not extensible. If we
want to do that, we will end up creating similar things for each of
them or re-creating index vacuum progress tracking feature while
creating a common infra. It might not be a problem as of now but I'm
concerned that introducing a feature that is not extensible and forces
users to allocate additional shmem might be a blocker in the future.
Looking at the precedent example, When we introduce the progress
tracking feature, we implemented it in an extensible way. On the other
hand, others in this thread seem to agree with this approach, so I'd
like to leave it to committers.
Anyway, here are some comments on v5-0001 patch:
+/* in commands/vacuumparallel.c */
+extern void VacuumWorkerProgressShmemInit(void);
+extern Size VacuumWorkerProgressShmemSize(void);
+extern void vacuum_worker_end(int leader_pid);
+extern void vacuum_worker_update(int leader_pid);
+extern void vacuum_worker_end_callback(int code, Datum arg);
+extern void set_vaccum_worker_progress(Datum *values);
These functions' body is not in vacuumparallel.c. As the comment says,
I think these functions should be implemented in vacuumparallel.c.
---
+/*
+ * set_vaccum_worker_progress --- updates the number of indexes that have been
+ * vacuumed or cleaned up in a parallel vacuum.
+ */
+void
+set_vaccum_worker_progress(Datum *values)
s/vaccum/vacuum/
---
+void
+set_vaccum_worker_progress(Datum *values)
+{
+ VacProgressEntry *entry;
+ int leader_pid = values[0];
I thik we should use DatumGetInt32().
---
+ entry = (VacProgressEntry *)
hash_search(VacuumWorkerProgressHash, &leader_pid, HASH_ENTER_NULL,
&found);
+
+ if (!entry)
+ elog(ERROR, "cannot allocate shared memory for vacuum
worker progress");
Since we raise an error in case of out of memory, I think we can use
HASH_ENTER instead of HASH_ENTER_NULL. Or do we want to emit a
detailed error message here?
---
+ VacuumWorkerProgressHash = NULL;
This line is not necessary.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-03-14 16:20 ` Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-14 16:20 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> I'm still unsure the current design of 0001 patch is better than other
> approaches we’ve discussed. Even users who don't use parallel vacuum
> are forced to allocate shared memory for index vacuum progress, with
> GetMaxBackends() entries from the beginning. Also, it’s likely to
> extend the progress tracking feature for other parallel operations in
> the future but I think the current design is not extensible. If we
> want to do that, we will end up creating similar things for each of
> them or re-creating index vacuum progress tracking feature while
> creating a common infra. It might not be a problem as of now but I'm
> concerned that introducing a feature that is not extensible and forces
> users to allocate additional shmem might be a blocker in the future.
> Looking at the precedent example, When we introduce the progress
> tracking feature, we implemented it in an extensible way. On the other
> hand, others in this thread seem to agree with this approach, so I'd
> like to leave it to committers.
Thanks for the review!
I think you make strong arguments as to why we need to take a different approach now than later.
Flaws with current patch set:
1. GetMaxBackends() is a really heavy-handed overallocation of a shared memory serving a very specific purpose.
2. Going with the approach of a vacuum specific hash breaks the design of progress which is meant to be extensible.
3. Even if we go with this current approach as an interim solution, it will be a real pain in the future.
With that said, v7 introduces the new infrastructure. 0001 includes the new infrastructure and 0002 takes advantage of this.
This approach is the following:
1. Introduces a new API called pgstat_progress_update_param_parallel along with some others support functions. This new infrastructure is in backend_progress.c
2. There is still a shared memory involved, but the size is capped to " max_worker_processes" which is the max to how many parallel workers can be doing work at any given time. The shared memory hash includes a st_progress_param array just like the Backend Status array.
typedef struct ProgressParallelEntry
{
pid_t leader_pid;
int64 st_progress_param[PGSTAT_NUM_PROGRESS_PARAM];
} ProgressParallelEntry;
3. The progress update function is "pgstat_progress_update_param_parallel" and will aggregate totals reported for a specific progress parameter
For example , it can be called lie below. In the case below, PROGRESS_VACUUM_INDEXES_COMPLETED is incremented by 1 in the shared memory entry shared by the workers and leader.
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
pgstat_progress_update_param_parallel(pvs->shared->leader_pid, PROGRESS_VACUUM_INDEXES_COMPLETED, 1); <<-----
break;
4. pg_stat_get_progress_info will call a function called pgstat_progress_set_parallel which will set the parameter value to the total from the shared memory hash.
I believe this approach gives proper infrastructure for future use-cases of workers reporting progress -and- does not do the heavy-handed shared memory allocation.
--
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v7-0001-Add-infrastructure-for-parallel-progress-reportin.patch (10.5K, ../../[email protected]/2-v7-0001-Add-infrastructure-for-parallel-progress-reportin.patch)
download | inline diff:
From 5314444d550893cf41a780c7a5cf63274e579dc7 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Mon, 14 Mar 2022 15:55:57 +0000
Subject: [PATCH v7 1/1] Add infrastructure for parallel progress reporting
Infrastructure to allow a parallel worker to report
progress. In a PARALLEL command, the workers and
leader can report progress using a new pgstat_progress
API.
The progress is maintaned in a shared memory hash
table for which progress values are aggregated for
all processes involved in the command.
pg_stat_get_progress_info reads from the shared memory
hash to report the aggregated data to the caller.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/5478DFCD-2333-401A-B2F0-0D186AB09228%40amazon.com
---
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/utils/activity/backend_progress.c | 166 ++++++++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 10 +-
src/include/commands/progress.h | 4 +
src/include/utils/backend_progress.h | 11 +-
6 files changed, 191 insertions(+), 4 deletions(-)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index cd4ebe2fc5..ccb9262b97 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -24,6 +24,7 @@
#include "access/twophase.h"
#include "access/xlogrecovery.h"
#include "commands/async.h"
+#include "commands/progress.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -145,6 +146,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, ProgressParallelShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -296,6 +298,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ ProgressParallelShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 6c7cf6c295..4212dea7f3 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+ProgressParallelLock 48
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index f29199725b..de69aaa8ad 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -10,10 +10,30 @@
*/
#include "postgres.h"
+#include "commands/progress.h"
#include "port/atomics.h" /* for memory barriers */
+#include "storage/ipc.h"
+#include "storage/lmgr.h"
+#include "storage/shmem.h"
#include "utils/backend_progress.h"
#include "utils/backend_status.h"
+/*
+ * Structs for parallel progress tracking.
+ *
+ * The parallel workers and leader report progress
+ * into a hash entry with a key of the leader pid.
+ */
+typedef struct ProgressParallelEntry
+{
+ pid_t leader_pid;
+ int64 st_progress_param[PGSTAT_NUM_PROGRESS_PARAM];
+} ProgressParallelEntry;
+
+static HTAB *ProgressParallelHash;
+
+/* We can only have as many parallel progress entries as max_parallel_workers */
+#define PROGRESS_PARALLEL_NUM_ENTRIES max_worker_processes
/*-----------
* pgstat_progress_start_command() -
@@ -110,3 +130,149 @@ pgstat_progress_end_command(void)
beentry->st_progress_command_target = InvalidOid;
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+
+/*-----------
+ * ProgressParallelShmemInit() -
+ *
+ * Initialize the parallel progress hash.
+ *-----------
+ */
+void
+ProgressParallelShmemInit(void)
+{
+ HASHCTL info;
+
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(ProgressParallelEntry);
+
+ ProgressParallelHash = ShmemInitHash("Parallel Progress hash",
+ PROGRESS_PARALLEL_NUM_ENTRIES,
+ PROGRESS_PARALLEL_NUM_ENTRIES,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+}
+
+/*-----------
+ * ProgressParallelShmemSize() -
+ *
+ * Calculate the size of the parallel progress hash.
+ *-----------
+ */
+Size
+ProgressParallelShmemSize(void)
+{
+ Size size = 0;
+
+ /* parallel progress hash table */
+ size = add_size(size, hash_estimate_size(PROGRESS_PARALLEL_NUM_ENTRIES,
+ sizeof(ProgressParallelEntry)));
+
+ return size;
+}
+
+/*-----------
+ * pgstat_progress_update_param_parallel() -
+ *
+ * Update the index'th member in then st_progress_param[] of the
+ * parallel progress hash table.
+ *-----------
+ */
+void
+pgstat_progress_update_param_parallel(int leader_pid, int index, int64 val)
+{
+ ProgressParallelEntry *entry;
+ bool found;
+
+ LWLockAcquire(ProgressParallelLock, LW_EXCLUSIVE);
+
+ entry = (ProgressParallelEntry *) hash_search(ProgressParallelHash, &leader_pid, HASH_ENTER, &found);
+
+ /*
+ * If the entry is not found, set the value for the index'th member,
+ * else increment the current value of the index'th member.
+ */
+ if (!found)
+ entry->st_progress_param[index] = val;
+ else
+ entry->st_progress_param[index] += val;
+
+ LWLockRelease(ProgressParallelLock);
+}
+
+/*-----------
+ * pgstat_progress_end_parallel() -
+ *
+ * This removes an entry with from the parallel progress
+ * hash table.
+ *-----------
+ */
+void
+pgstat_progress_end_parallel(int leader_pid)
+{
+ LWLockAcquire(ProgressParallelLock, LW_EXCLUSIVE);
+
+ /*
+ * Remove from hashtable. It should always be present,
+ * but don't complain if it's not.
+ */
+ hash_search(ProgressParallelHash, &leader_pid, HASH_REMOVE, NULL);
+
+ LWLockRelease(ProgressParallelLock);
+}
+
+/*-----------
+ * pgstat_progress_end_parallel_callback() -
+ *
+ * PG_ENSURE_ERROR_CLEANUP callback. The caller is responsible
+ * for ensuring cleanup when invoking pgstat_progress_update_param_parallel.
+ *-----------
+ */
+void
+pgstat_progress_end_parallel_callback(int code, Datum arg)
+{
+ pgstat_progress_end_parallel(DatumGetInt32(arg));
+}
+
+/*-----------
+ * pgstat_progress_set_parallel() -
+ *
+ * This routine is called by pg_stat_get_progress_info
+ * to update the datum with values from the parallel progress
+ * hash.
+ *-----------
+ */
+void
+pgstat_progress_set_parallel(Datum *values)
+{
+ ProgressParallelEntry *entry;
+ /* First element of the datum is always the pid */
+ int leader_pid = values[0];
+
+ LWLockAcquire(ProgressParallelLock, LW_SHARED);
+
+ entry = (ProgressParallelEntry *) hash_search(ProgressParallelHash, &leader_pid, HASH_FIND, NULL);
+
+ /*
+ * If an entry is not found, it is because we looked at a pid that is not involved in a parallel command,
+ * therefore release the read lock and break.
+ */
+ if (!entry)
+ {
+ LWLockRelease(ProgressParallelLock);
+ return;
+ }
+
+ for (int i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
+ {
+ int64 val = entry->st_progress_param[i];
+
+ /*
+ * We only care about hash entry members that have been updated by
+ * parallel workers ( or leader ). This is true if the member's value > 0.
+ */
+ if (val > 0)
+ values[i + PGSTAT_NUM_PROGRESS_COMMON] = val;
+ }
+
+ LWLockRelease(ProgressParallelLock);
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index eff45b16f2..1856e9c3b6 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -18,6 +18,7 @@
#include "access/xlog.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/progress.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -456,7 +457,7 @@ pg_stat_get_backend_idset(PG_FUNCTION_ARGS)
Datum
pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + 3
+#define PG_STAT_GET_PROGRESS_COLS PGSTAT_NUM_PROGRESS_PARAM + PGSTAT_NUM_PROGRESS_COMMON
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
char *cmd = text_to_cstring(PG_GETARG_TEXT_PP(0));
@@ -518,15 +519,18 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
{
values[2] = ObjectIdGetDatum(beentry->st_progress_command_target);
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- values[i + 3] = Int64GetDatum(beentry->st_progress_param[i]);
+ values[i + PGSTAT_NUM_PROGRESS_COMMON] = Int64GetDatum(beentry->st_progress_param[i]);
}
else
{
nulls[2] = true;
for (i = 0; i < PGSTAT_NUM_PROGRESS_PARAM; i++)
- nulls[i + 3] = true;
+ nulls[i + PGSTAT_NUM_PROGRESS_COMMON] = true;
}
+ /* Before returning the datum, set the fields from parallel progress tracking */
+ pgstat_progress_set_parallel(values);
+
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..cd3122c344 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_TOTAL_INDEXES 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
@@ -151,4 +153,6 @@
#define PROGRESS_COPY_TYPE_PIPE 3
#define PROGRESS_COPY_TYPE_CALLBACK 4
+/* Number of common fields at the start of progress views */
+#define PGSTAT_NUM_PROGRESS_COMMON 3
#endif
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..291d9413c7 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -39,6 +39,15 @@ extern void pgstat_progress_update_param(int index, int64 val);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
-
+/* -----------
+ * Routines for parallel command progress reporting
+ * -----------
+ */
+extern void ProgressParallelShmemInit(void);
+extern Size ProgressParallelShmemSize(void);
+extern void pgstat_progress_update_param_parallel(int leader_pid, int index, int64 val);
+extern void pgstat_progress_end_parallel(int leader_pid);
+extern void pgstat_progress_end_parallel_callback(int code, Datum arg);
+extern void pgstat_progress_set_parallel(Datum *values);
#endif /* BACKEND_PROGRESS_H */
--
2.32.0
[application/octet-stream] v7-0002-Show-progress-for-index-vacuums.patch (10.1K, ../../[email protected]/3-v7-0002-Show-progress-for-index-vacuums.patch)
download | inline diff:
From 5b49df7ce65893c19008376c4cbab7e9fc5eca33 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Mon, 14 Mar 2022 14:07:31 +0000
Subject: [PATCH v7 2/4] Show progress for index vacuums
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 25 ++++++++++++
src/backend/access/heap/vacuumlazy.c | 59 +++++++++++++++++++++++----
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 7 ++++
src/test/regress/expected/rules.out | 4 +-
5 files changed, 89 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 9fb62fec8e..1acc741da9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6227,6 +6227,31 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when there are no indexes to process
+ or when failsafe is triggered during the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_processed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes already processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 87ab7775ae..65e4440dc3 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -58,6 +58,7 @@
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/freespace.h"
+#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "tcop/tcopprot.h"
#include "utils/lsyscache.h"
@@ -2313,9 +2314,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/* Report that we are now vacuuming indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
+ /* Advertise the number of indexes we are vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
+ int indexes_processed = 1;
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
@@ -2325,6 +2330,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_processed++);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2335,9 +2342,21 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
- vacrel->num_index_scans);
+ /*
+ * Outsource everything to parallel variant
+ *
+ * To ensure cleanup of the progress worker hash entry,
+ * wrap parallel_vacuum_bulkdel_all_indexes in a
+ * PG_ENSURE_ERROR_CLEANUP
+ *
+ */
+ PG_ENSURE_ERROR_CLEANUP(pgstat_progress_end_parallel_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, vacrel->old_live_tuples,
+ vacrel->num_index_scans);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(pgstat_progress_end_parallel_callback, Int32GetDatum(MyProcPid));
+ pgstat_progress_end_parallel(MyProcPid);
/*
* Do a postcheck to consider applying wraparound failsafe now. Note
@@ -2345,6 +2364,9 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
*/
if (lazy_check_wraparound_failsafe(vacrel))
allindexes = false;
+
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
}
/*
@@ -2628,6 +2650,8 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2664,9 +2688,13 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
/* Report that we are now cleaning up indexes */
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
+ /* Advertise the number of indexes we are cleaning up */
+ pgstat_progress_update_param(PROGRESS_VACUUM_TOTAL_INDEXES, vacrel->nindexes);
if (!ParallelVacuumIsActive(vacrel))
{
+ int indexes_processed = 1;
+
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
@@ -2675,15 +2703,32 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_processed++);
}
}
else
{
- /* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, reltuples,
- vacrel->num_index_scans,
- estimated_count);
+ /*
+ * Outsource everything to parallel variant
+ *
+ * To ensure cleanup of the paralle progress hash entry,
+ * wrap parallel_vacuum_bulkdel_all_indexes in a
+ * PG_ENSURE_ERROR_CLEANUP
+ *
+ */
+ PG_ENSURE_ERROR_CLEANUP(pgstat_progress_end_parallel_callback, Int32GetDatum(MyProcPid));
+ {
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, reltuples,
+ vacrel->num_index_scans,
+ estimated_count);
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(pgstat_progress_end_parallel_callback, Int32GetDatum(MyProcPid));
+ pgstat_progress_end_parallel(MyProcPid);
}
+
+ /* reset index progress */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index bb1ac30cd1..35ea25026f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1126,7 +1126,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 974a29e7a9..b491728425 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -29,6 +29,7 @@
#include "access/amapi.h"
#include "access/table.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -101,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Leader PID of the vacuum */
+ int leader_pid;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -357,6 +361,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->leader_pid = MyProcPid;
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -844,9 +849,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
+ pgstat_progress_update_param_parallel(pvs->shared->leader_pid, PROGRESS_VACUUM_INDEXES_COMPLETED, 1);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
istat_res = vac_cleanup_one_index(&ivinfo, istat);
+ pgstat_progress_update_param_parallel(pvs->shared->leader_pid, PROGRESS_VACUUM_INDEXES_COMPLETED, 1);
break;
default:
elog(ERROR, "unexpected parallel vacuum index status %d for index \"%s\"",
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac468568a1..d70a176514 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,7 +2002,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_replication| SELECT s.pid,
--
2.32.0
[application/octet-stream] v7-0003-Expose-indexes-being-processed-in-a-VACUUM-operat.patch (16.8K, ../../[email protected]/4-v7-0003-Expose-indexes-being-processed-in-a-VACUUM-operat.patch)
download | inline diff:
From 7a5244805d4694a74f073a89b2e808310810bbd1 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Mon, 14 Mar 2022 14:26:20 +0000
Subject: [PATCH v7 3/4] Expose indexes being processed in a VACUUM operation.
A new view called pg_stat_progress_vacuum_index to show the indexrelid
being vacuumed or cleaned during a vacuum. The view also shows the
number of tuples removed for the index during the vacuuming indexes phase.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
---
doc/src/sgml/monitoring.sgml | 108 ++++++++++++++++++++++++++
src/backend/access/gin/ginvacuum.c | 3 +
src/backend/access/gist/gistvacuum.c | 3 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/vacuumlazy.c | 14 ++++
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgvacuum.c | 4 +
src/backend/catalog/system_views.sql | 18 ++++-
src/backend/commands/vacuumparallel.c | 12 +++
src/include/commands/progress.h | 3 +
src/test/regress/expected/rules.out | 17 +++-
11 files changed, 182 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1acc741da9..240d2438d5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -379,6 +379,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_progress_vacuum_index</structname><indexterm><primary>pg_stat_progress_vacuum_index</primary></indexterm></entry>
+ <entry>One row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum, showing current progress.
+ See <xref linkend='vacuum-progress-reporting'/>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_cluster</structname><indexterm><primary>pg_stat_progress_cluster</primary></indexterm></entry>
<entry>One row for each backend running
@@ -6256,6 +6265,105 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</tgroup>
</table>
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) that is
+ currently performing the <literal>vacuuming indexes</literal> or
+ <literal>cleaning up indexes</literal> phase of the vacuum.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-index-view" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the index being processed in the ongoing phase of the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the parallel vacuum leader, if this process is a
+ parallel vacuum worker. <literal>NULL</literal> if this process is a
+ parallel vacuum leader or does not participate in parallel vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase of a vacuum. Only the
+ <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase will be listed in this view. See <xref linkend="vacuum-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>tuples_removed</structfield> <type>oid</type>
+ </para>
+ <para>
+ The number of index tuples removed by the <literal>vacuuming indexes</literal> phase.
+ This field is <literal>0</literal> during the <literal>cleaning up indexes</literal>
+ phase.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
<table id="vacuum-phases">
<title>VACUUM Phases</title>
<tgroup cols="2">
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..1d5d003780 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,8 +17,10 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -60,6 +62,7 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
if (gvs->callback(items + i, gvs->callback_state))
{
gvs->result->tuples_removed += 1;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, gvs->result->tuples_removed);
if (!tmpitems)
{
/*
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index aac4afab8f..8a0f23388b 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,9 +17,11 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "utils/memutils.h"
@@ -375,6 +377,7 @@ restart:
END_CRIT_SECTION();
vstate->stats->tuples_removed += ntodelete;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, vstate->stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index a259a301fa..23dacee52e 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -632,6 +632,7 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* hashvacuumcleanup will fill in num_pages */
return stats;
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 65e4440dc3..f33faabbe9 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -348,6 +348,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
/*
* Get OldestXmin cutoff, which is used to determine which deleted tuples
@@ -2326,12 +2327,19 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being vacuumed in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_processed++);
+ /* Advertise we are done vacuuming indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2700,11 +2708,17 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* Advertise the index being cleaned in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_processed++);
+
+ /* Advertise we are done cleaning indexes in non-parallel vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
}
}
else
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index c9b4964c1e..09edf49082 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1273,6 +1273,7 @@ backtrack:
nupdatable);
stats->tuples_removed += nhtidsdead;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, stats->tuples_removed);
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..db73f8ef59 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,8 +21,10 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
@@ -160,6 +162,7 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
bds->stats->tuples_removed += 1;
deletable[i] = true;
nDeletable++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
@@ -430,6 +433,7 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
xlrec.nDelete++;
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, bds->stats->tuples_removed);
}
else
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 35ea25026f..88375aad0a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1129,7 +1129,23 @@ CREATE VIEW pg_stat_progress_vacuum AS
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
- LEFT JOIN pg_database D ON S.datid = D.oid;
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.pid = S.param12;
+
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid AS pid,
+ S.datid AS datid,
+ D.datname AS datname,
+ S.param10 AS indexrelid,
+ S.param12 AS leader_pid,
+ CASE S.param1 WHEN 2 THEN 'vacuuming indexes'
+ WHEN 4 THEN 'cleaning up indexes'
+ END AS phase,
+ S.param11 AS tuples_removed
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 IN (2, 4) AND S.param10 > 0;
CREATE VIEW pg_stat_progress_cluster AS
SELECT
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index b491728425..a8a1f221e5 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -845,13 +845,18 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
+ /* Advertise the index we are cleaning or vacuuming */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, RelationGetRelid(indrel));
+
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
pgstat_progress_update_param_parallel(pvs->shared->leader_pid, PROGRESS_VACUUM_INDEXES_COMPLETED, 1);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
pgstat_progress_update_param_parallel(pvs->shared->leader_pid, PROGRESS_VACUUM_INDEXES_COMPLETED, 1);
break;
@@ -888,6 +893,10 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
*/
indstats->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+ /* Advertise we are no longer vacuuming/cleaning an index */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXRELID, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_TUPLES_REMOVED, 0);
+
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
@@ -972,6 +981,8 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
* workers.
*/
rel = table_open(shared->relid, ShareUpdateExclusiveLock);
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM, RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, shared->leader_pid);
/*
* Open all indexes. indrels are sorted in order by OID, which should be
@@ -1042,6 +1053,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
vac_close_indexes(nindexes, indrels, RowExclusiveLock);
table_close(rel, ShareUpdateExclusiveLock);
+ pgstat_progress_end_command();
FreeAccessStrategy(pvs.bstrategy);
}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index cd3122c344..879691b37d 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -27,6 +27,9 @@
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
#define PROGRESS_VACUUM_TOTAL_INDEXES 7
#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
+#define PROGRESS_VACUUM_INDEXRELID 9
+#define PROGRESS_VACUUM_TUPLES_REMOVED 10
+#define PROGRESS_VACUUM_LEADER_PID 11
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d70a176514..d2c66bf034 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2006,7 +2006,22 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param8 AS indexes_total,
s.param9 AS indexes_processed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
- LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE (s.pid = s.param12);
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.param10 AS indexrelid,
+ s.param12 AS leader_pid,
+ CASE s.param1
+ WHEN 2 THEN 'vacuuming indexes'::text
+ WHEN 4 THEN 'cleaning up indexes'::text
+ ELSE NULL::text
+ END AS phase,
+ s.param11 AS tuples_removed
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE ((s.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint])) AND (s.param10 > 0));
pg_stat_replication| SELECT s.pid,
s.usesysid,
u.rolname AS usename,
--
2.32.0
[application/octet-stream] v7-0004-Rename-index_vacuum_count-in-pg_stat_pogress_vacu.patch (2.8K, ../../[email protected]/5-v7-0004-Rename-index_vacuum_count-in-pg_stat_pogress_vacu.patch)
download | inline diff:
From c2f2074b595219d2063822eae1ea0ca87a5bd205 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Mon, 14 Mar 2022 14:27:57 +0000
Subject: [PATCH v7 4/4] Rename "index_vacuum_count" in pg_stat_pogress_vacuum.
Commit 76d8a1bfd5207d28a4e9fe98a0e1ea7c096d70aa introduces 2 new columns
to track index vacuum/cleanup progress. The columns have "index" in the
name. To make it clear what the existing "index_vacuum_count" column
refers to, which is the cycle count of the index vacuum, this change
renames the column to "index_vacuum_count"
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Justin Pryzby
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/catalog/system_views.sql | 2 +-
src/test/regress/expected/rules.out | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 240d2438d5..1e5af0b400 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6210,7 +6210,7 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>index_vacuum_count</structfield> <type>bigint</type>
+ <structfield>index_vacuum_cycle_count</structfield> <type>bigint</type>
</para>
<para>
Number of completed index vacuum cycles.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 88375aad0a..639ad00a71 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
WHEN 6 THEN 'performing final cleanup'
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
- S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
+ S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_cycle_count,
S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
S.param8 AS indexes_total, S.param9 AS indexes_processed
FROM pg_stat_get_progress_info('VACUUM') AS S
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d2c66bf034..7a6bf3af27 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2000,7 +2000,7 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param2 AS heap_blks_total,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
- s.param5 AS index_vacuum_count,
+ s.param5 AS index_vacuum_cycle_count,
s.param6 AS max_dead_tuples,
s.param7 AS num_dead_tuples,
s.param8 AS indexes_total,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-22 03:19 ` Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-03-22 03:19 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Sorry for the late reply.
On Tue, Mar 15, 2022 at 1:20 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> > I'm still unsure the current design of 0001 patch is better than other
> > approaches we’ve discussed. Even users who don't use parallel vacuum
> > are forced to allocate shared memory for index vacuum progress, with
> > GetMaxBackends() entries from the beginning. Also, it’s likely to
> > extend the progress tracking feature for other parallel operations in
> > the future but I think the current design is not extensible. If we
> > want to do that, we will end up creating similar things for each of
> > them or re-creating index vacuum progress tracking feature while
> > creating a common infra. It might not be a problem as of now but I'm
> > concerned that introducing a feature that is not extensible and forces
> > users to allocate additional shmem might be a blocker in the future.
> > Looking at the precedent example, When we introduce the progress
> > tracking feature, we implemented it in an extensible way. On the other
> > hand, others in this thread seem to agree with this approach, so I'd
> > like to leave it to committers.
>
> Thanks for the review!
>
> I think you make strong arguments as to why we need to take a different approach now than later.
>
> Flaws with current patch set:
>
> 1. GetMaxBackends() is a really heavy-handed overallocation of a shared memory serving a very specific purpose.
> 2. Going with the approach of a vacuum specific hash breaks the design of progress which is meant to be extensible.
> 3. Even if we go with this current approach as an interim solution, it will be a real pain in the future.
>
> With that said, v7 introduces the new infrastructure. 0001 includes the new infrastructure and 0002 takes advantage of this.
>
> This approach is the following:
>
> 1. Introduces a new API called pgstat_progress_update_param_parallel along with some others support functions. This new infrastructure is in backend_progress.c
>
> 2. There is still a shared memory involved, but the size is capped to " max_worker_processes" which is the max to how many parallel workers can be doing work at any given time. The shared memory hash includes a st_progress_param array just like the Backend Status array.
I think that there is a corner case where a parallel operation could
not perform due to the lack of a free shared hash entry, because there
is a window between a parallel worker exiting and the leader
deallocating the hash table entry.
BTW have we discussed another idea I mentioned before that we have the
leader process periodically check the number of completed indexes and
advertise it in its progress information? I'm not sure which one is
better but this idea would require only changes of vacuum code and
probably simpler than the current idea.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-03-22 07:27 ` Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-22 07:27 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> BTW have we discussed another idea I mentioned before that we have the
> leader process periodically check the number of completed indexes and
> advertise it in its progress information? I'm not sure which one is
> better but this idea would require only changes of vacuum code and
> probably simpler than the current idea.
> Regards,
If I understand correctly, to accomplish this we will need to have the leader
check the number of indexes completed In the ambukdelete or amvacuumcleanup
callbacks. These routines do not know about PVIndStats, and they are called
by both parallel and non-parallel vacuums.
From what I can see, PVIndstats will need to be passed down to these routines
or pass a NULL for non-parallel vacuums.
Sami
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-22 07:48 ` Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-03-22 07:48 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Tue, Mar 22, 2022 at 4:27 PM Imseih (AWS), Sami <[email protected]> wrote:
>
> > BTW have we discussed another idea I mentioned before that we have the
> > leader process periodically check the number of completed indexes and
> > advertise it in its progress information? I'm not sure which one is
> > better but this idea would require only changes of vacuum code and
> > probably simpler than the current idea.
>
> > Regards,
>
>
> If I understand correctly, to accomplish this we will need to have the leader
> check the number of indexes completed In the ambukdelete or amvacuumcleanup
> callbacks. These routines do not know about PVIndStats, and they are called
> by both parallel and non-parallel vacuums.
>
> From what I can see, PVIndstats will need to be passed down to these routines
> or pass a NULL for non-parallel vacuums.
>
Can the leader pass a callback that checks PVIndStats to ambulkdelete
an amvacuumcleanup callbacks? I think that in the passed callback, the
leader checks if the number of processed indexes and updates its
progress information if the current progress needs to be updated.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-03-22 21:57 ` Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-22 21:57 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> Can the leader pass a callback that checks PVIndStats to ambulkdelete
> an amvacuumcleanup callbacks? I think that in the passed callback, the
> leader checks if the number of processed indexes and updates its
> progress information if the current progress needs to be updated.
Thanks for the suggestion.
I looked at this option a but today and found that passing the callback
will also require signature changes to the ambulkdelete and
amvacuumcleanup routines.
This will also require us to check after x pages have been
scanned inside vacuumscan and vacuumcleanup. After x pages
the callback can then update the leaders progress.
I am not sure if adding additional complexity to the scan/cleanup path
is justified for what this patch is attempting to do.
There will also be a lag of the leader updating the progress as it
must scan x amount of pages before updating. Obviously, the more
Pages to the scan, the longer the lag will be.
Would like to hear your thoughts on the above.
Regards,
Sami Imseih
Amazon Web Services.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-25 14:54 ` Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-03-25 14:54 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Wed, Mar 23, 2022 at 6:57 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> > Can the leader pass a callback that checks PVIndStats to ambulkdelete
> > an amvacuumcleanup callbacks? I think that in the passed callback, the
> > leader checks if the number of processed indexes and updates its
> > progress information if the current progress needs to be updated.
>
> Thanks for the suggestion.
>
> I looked at this option a but today and found that passing the callback
> will also require signature changes to the ambulkdelete and
> amvacuumcleanup routines.
I think it would not be a critical problem since it's a new feature.
>
> This will also require us to check after x pages have been
> scanned inside vacuumscan and vacuumcleanup. After x pages
> the callback can then update the leaders progress.
> I am not sure if adding additional complexity to the scan/cleanup path
> is justified for what this patch is attempting to do.
>
> There will also be a lag of the leader updating the progress as it
> must scan x amount of pages before updating. Obviously, the more
> Pages to the scan, the longer the lag will be.
Fair points.
On the other hand, the approach of the current patch requires more
memory for progress tracking, which could fail, e.g., due to running
out of hashtable entries. I think that it would be worse that the
parallel operation failed to start due to not being able to track the
progress than the above concerns you mentioned such as introducing
additional complexity and a possible lag of progress updates. So if we
go with the current approach, I think we need to make sure enough (and
not too many) hash table entries.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-03-29 12:08 ` Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-29 12:08 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Sorry for the late reply.
> additional complexity and a possible lag of progress updates. So if we
> go with the current approach, I think we need to make sure enough (and
> not too many) hash table entries.
The hash table can be set 4 times the size of
max_worker_processes which should give more than
enough padding.
Note that max_parallel_maintenance_workers
is what should be used, but since it's dynamic, it cannot
be used to determine the size of shared memory.
Regards,
---
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-03-29 12:25 ` Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-03-29 12:25 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>
> I think that's an absolute no-go. Adding locking to progress reporting,
> particularly a single central lwlock, is going to *vastly* increase the
> overhead incurred by progress reporting.
Sorry for the late reply.
The usage of the shared memory will be limited
to PARALLEL maintenance operations. For now,
it will only be populated for parallel vacuums.
Autovacuum for example will not be required to
populate this shared memory.
Regards,
---
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-04-03 16:50 ` Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Andres Freund @ 2022-04-03 16:50 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>
Hi,
On 2022-03-29 12:25:52 +0000, Imseih (AWS), Sami wrote:
> > I think that's an absolute no-go. Adding locking to progress reporting,
> > particularly a single central lwlock, is going to *vastly* increase the
> > overhead incurred by progress reporting.
>
> Sorry for the late reply.
>
> The usage of the shared memory will be limited
> to PARALLEL maintenance operations. For now,
> it will only be populated for parallel vacuums.
> Autovacuum for example will not be required to
> populate this shared memory.
I nevertheless think that's not acceptable. The whole premise of the progress
reporting infrastructure is to be low overhead. It's OK to require locking to
initialize parallel progress reporting, it's definitely not ok to require
locking to report progress.
Leaving the locking aside, doing a hashtable lookup for each progress report
is pretty expensive.
Why isn't the obvious thing to do here to provide a way to associate workers
with their leaders in shared memory, but to use the existing progress fields
to report progress? Then, when querying progress, the leader and workers
progress fields can be combined to show the overall progress?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
@ 2022-04-05 16:42 ` Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-05 17:31 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-04-05 16:42 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>
> I nevertheless think that's not acceptable. The whole premise of the progress
> reporting infrastructure is to be low overhead. It's OK to require locking to
> initialize parallel progress reporting, it's definitely not ok to require
> locking to report progress.
Fair point.
> Why isn't the obvious thing to do here to provide a way to associate workers
> with their leaders in shared memory, but to use the existing progress fields
> to report progress? Then, when querying progress, the leader and workers
> progress fields can be combined to show the overall progress?
The original intent was this, however the workers
can exit before the command completes and the
worker progress data will be lost.
This is why the shared memory was introduced.
This allows the worker progress to persist for the duration
of the command.
Regards,
Sami Imseih
Amazon Web Services.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-04-05 17:19 ` Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Robert Haas @ 2022-04-05 17:19 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Andres Freund <[email protected]>; Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>
On Tue, Apr 5, 2022 at 12:42 PM Imseih (AWS), Sami <[email protected]> wrote:
> > Why isn't the obvious thing to do here to provide a way to associate workers
> > with their leaders in shared memory, but to use the existing progress fields
> > to report progress? Then, when querying progress, the leader and workers
> > progress fields can be combined to show the overall progress?
>
> The original intent was this, however the workers
> can exit before the command completes and the
> worker progress data will be lost.
> This is why the shared memory was introduced.
> This allows the worker progress to persist for the duration
> of the command.
At the beginning of a parallel operation, we allocate a chunk of
dynamic shared memory which persists even after some or all workers
have exited. It's only torn down at the end of the parallel operation.
That seems like the appropriate place to be storing any kind of data
that needs to be propagated between parallel workers. The current
patch uses the main shared memory segment, which seems unacceptable to
me.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
@ 2022-04-06 21:22 ` Imseih (AWS), Sami <[email protected]>
2022-04-07 13:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-04-06 21:22 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>
> At the beginning of a parallel operation, we allocate a chunk of>
> dynamic shared memory which persists even after some or all workers
> have exited. It's only torn down at the end of the parallel operation.
> That seems like the appropriate place to be storing any kind of data
> that needs to be propagated between parallel workers. The current
> patch uses the main shared memory segment, which seems unacceptable to
> me.
Correct, DSM does track shared data. However only participating
processes in the parallel vacuum can attach and lookup this data.
The purpose of the main shared memory is to allow a process that
Is querying the progress views to retrieve the information.
Regards,
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-04-07 13:20 ` Robert Haas <[email protected]>
2022-04-07 15:38 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Robert Haas @ 2022-04-07 13:20 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Andres Freund <[email protected]>; Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>
On Wed, Apr 6, 2022 at 5:22 PM Imseih (AWS), Sami <[email protected]> wrote:
> > At the beginning of a parallel operation, we allocate a chunk of>
> > dynamic shared memory which persists even after some or all workers
> > have exited. It's only torn down at the end of the parallel operation.
> > That seems like the appropriate place to be storing any kind of data
> > that needs to be propagated between parallel workers. The current
> > patch uses the main shared memory segment, which seems unacceptable to
> > me.
>
> Correct, DSM does track shared data. However only participating
> processes in the parallel vacuum can attach and lookup this data.
>
> The purpose of the main shared memory is to allow a process that
> Is querying the progress views to retrieve the information.
Sure, but I think that you should likely be doing what Andres
recommended before:
# Why isn't the obvious thing to do here to provide a way to associate workers
# with their leaders in shared memory, but to use the existing progress fields
# to report progress? Then, when querying progress, the leader and workers
# progress fields can be combined to show the overall progress?
That is, I am imagining that you would want to use DSM to propagate
data from workers back to the leader and then have the leader report
the data using the existing progress-reporting facilities. Now, if we
really need a whole row from each worker that doesn't work, but why do
we need that?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-07 13:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
@ 2022-04-07 15:38 ` Masahiko Sawada <[email protected]>
2022-04-07 23:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Greg Stark <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-04-07 15:38 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Imseih (AWS), Sami <[email protected]>; Andres Freund <[email protected]>; Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Thu, Apr 7, 2022 at 10:20 PM Robert Haas <[email protected]> wrote:
>
> On Wed, Apr 6, 2022 at 5:22 PM Imseih (AWS), Sami <[email protected]> wrote:
> > > At the beginning of a parallel operation, we allocate a chunk of>
> > > dynamic shared memory which persists even after some or all workers
> > > have exited. It's only torn down at the end of the parallel operation.
> > > That seems like the appropriate place to be storing any kind of data
> > > that needs to be propagated between parallel workers. The current
> > > patch uses the main shared memory segment, which seems unacceptable to
> > > me.
> >
> > Correct, DSM does track shared data. However only participating
> > processes in the parallel vacuum can attach and lookup this data.
> >
> > The purpose of the main shared memory is to allow a process that
> > Is querying the progress views to retrieve the information.
>
> Sure, but I think that you should likely be doing what Andres
> recommended before:
>
> # Why isn't the obvious thing to do here to provide a way to associate workers
> # with their leaders in shared memory, but to use the existing progress fields
> # to report progress? Then, when querying progress, the leader and workers
> # progress fields can be combined to show the overall progress?
>
> That is, I am imagining that you would want to use DSM to propagate
> data from workers back to the leader and then have the leader report
> the data using the existing progress-reporting facilities. Now, if we
> really need a whole row from each worker that doesn't work, but why do
> we need that?
+1
I also proposed the same idea before[1]. The leader can know how many
indexes are processed so far by checking PVIndStats.status allocated
on DSM for each index. We can have the leader check it and update the
progress information before and after vacuuming one index. If we want
to update the progress information more timely, probably we can pass a
callback function to ambulkdelete and amvacuumcleanup so that the
leader can do that periodically, e.g., every 1000 blocks, while
vacuuming an index.
Regards,
[1] https://www.postgresql.org/message-id/CAD21AoBW6SMJ96CNoMeu%2Bf_BR4jmatPcfVA016FdD2hkLDsaTA%40mail.g...
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-07 13:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-07 15:38 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-04-07 23:25 ` Greg Stark <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Greg Stark @ 2022-04-07 23:25 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Robert Haas <[email protected]>; Imseih (AWS), Sami <[email protected]>; Andres Freund <[email protected]>; Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
It looks like this patch got feedback from Andres and Robert with some
significant design change recommendations. I'm marking the patch
Returned with Feedback. Feel free to add it back to a future
commitfest when a new version is ready.
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-04-14 01:32 ` Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-04-14 01:32 UTC (permalink / raw)
To: Robert Haas <[email protected]>; Masahiko Sawada <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Taking the above feedback, I modified the patches
and I believe this approach should be acceptable.
For now, I also reduced the scope to only exposing
Indexes_total and indexes_completed in
pg_stat_progress_vacuum. I will create a new CF entry
for the new view pg_stat_progress_vacuum_index.
V10-0001: This patch adds a callback to ParallelContext
that could be called by the leader in vacuumparallel.c
and more importantly while the leader is waiting
for the parallel workers to complete inside.
This ensures that the leader is continuously polling and
reporting completed indexes for the life of the PARALLEL
VACUUM. This covers cases where the leader completes
vacuuming before the workers complete.
V10-0002: This implements the indexes_total and
indexes_completed columns in pg_stat_progress_vacuum.
This work is now tracked in the next commitfest:
https://commitfest.postgresql.org/38/3617/
Regards,
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v10-0001-Add-progress-reporting-callback-to-ParallelConte.patch (2.3K, ../../[email protected]/2-v10-0001-Add-progress-reporting-callback-to-ParallelConte.patch)
download | inline diff:
From 94fc3655b55cff3346a8afdf0bac2ee9b26fd035 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 14 Apr 2022 00:06:14 +0000
Subject: [PATCH v10 1/2] Add progress reporting callback to ParallelContext
The purpose of supporting a progress reporting
callback in ParallelContext is to allow for the
leader process to report progress while waiting
for workers to complete.
The first use-case for this is to report index
progress in pg_stat_progress_vacuum.
---
src/backend/access/transam/parallel.c | 16 ++++++++++++++++
src/include/access/parallel.h | 5 +++++
2 files changed, 21 insertions(+)
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index df0cd77558..bfe3275d8b 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -774,6 +774,22 @@ WaitForParallelWorkersToFinish(ParallelContext *pcxt)
*/
CHECK_FOR_INTERRUPTS();
+ /*
+ * We call the parallel progress callback while
+ * waiting for the parallel workers to finish.
+ * This is to ensure that the leader keeps
+ * updating progress when waiting for
+ * parallel workers to finish.
+ *
+ * We must ensure that pcxt->parallel_progress_callback
+ * is set before calling as not all parallel
+ * operations will set a callback.
+ */
+ if (pcxt->parallel_progress_callback)
+ {
+ pcxt->parallel_progress_callback(pcxt->parallel_progress_callback_arg);
+ }
+
for (i = 0; i < pcxt->nworkers_launched; ++i)
{
/*
diff --git a/src/include/access/parallel.h b/src/include/access/parallel.h
index 983841d45e..53a3d13ec0 100644
--- a/src/include/access/parallel.h
+++ b/src/include/access/parallel.h
@@ -20,6 +20,9 @@
#include "storage/shm_mq.h"
#include "storage/shm_toc.h"
+/* progress callback definition */
+typedef void (*ParallelProgressCallback) (void *parallel_progress_callback_state);
+
typedef void (*parallel_worker_main_type) (dsm_segment *seg, shm_toc *toc);
typedef struct ParallelWorkerInfo
@@ -46,6 +49,8 @@ typedef struct ParallelContext
ParallelWorkerInfo *worker;
int nknown_attached_workers;
bool *known_attached_workers;
+ ParallelProgressCallback parallel_progress_callback;
+ void *parallel_progress_callback_arg;
} ParallelContext;
typedef struct ParallelWorkerContext
--
2.32.0
[application/octet-stream] v10-0002-Show-progress-for-index-vacuums.patch (9.6K, ../../[email protected]/3-v10-0002-Show-progress-for-index-vacuums.patch)
download | inline diff:
From e5f70a1853c8be8b02144420f84763ccfcee9b13 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Thu, 14 Apr 2022 00:47:42 +0000
Subject: [PATCH 2/2] Show progress for index vacuums
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 25 ++++++++++++++++
src/backend/access/heap/vacuumlazy.c | 19 ++++++++++++
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 42 +++++++++++++++++++++++++++
src/include/commands/progress.h | 2 ++
src/test/regress/expected/rules.out | 4 ++-
6 files changed, 93 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 180798a6b9..c3e60e57d9 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6396,6 +6396,31 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when there are no indexes to process
+ or when failsafe is triggered during the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_completed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes already processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 788db569b2..587aa668f9 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -415,6 +415,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+ /* Report the number of indexes to vacuum/cleanup */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2327,6 +2330,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /* Report the number of indexes vacuumed */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
if (lazy_check_wraparound_failsafe(vacrel))
{
@@ -2361,6 +2366,10 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
vacrel->dead_items->num_items == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
+ /* Report that we're done vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
/*
* Increase and report the number of index scans.
*
@@ -2625,6 +2634,10 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ /* Report that we're no longer vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
get_database_name(MyDatabaseId),
@@ -2671,6 +2684,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+ /* Report the number of indexes cleaned */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
}
}
else
@@ -2680,6 +2695,10 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->num_index_scans,
estimated_count);
}
+
+ /* Report that we're done cleaning indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 0fc614e32c..bbd2d6352b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1155,7 +1155,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_completed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index bbf3b69c57..238c49d7d1 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -30,6 +30,7 @@
#include "access/table.h"
#include "access/xact.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -213,6 +214,7 @@ static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation
static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_index_scans,
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
+static void parallel_vacuum_progress_callback(void *arg);
/*
* Try to enter parallel mode and create a parallel context. Then initialize
@@ -288,6 +290,10 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
+ /* Setup the Parallel Progress Callback */
+ pvs->pcxt->parallel_progress_callback = parallel_vacuum_progress_callback;
+ pvs->pcxt->parallel_progress_callback_arg = pvs;
+
/*
* Estimate space for BufferUsage and WalUsage --
* PARALLEL_VACUUM_KEY_BUFFER_USAGE and PARALLEL_VACUUM_KEY_WAL_USAGE.
@@ -856,6 +862,18 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
RelationGetRelationName(indrel));
}
+ /*
+ * If we are the leader, update index vacuum progress.
+ *
+ * For convenience, rely on the parallel_progress_callback
+ * which calculates and reports the number of indexes
+ * vacuumed/cleaned up in parallel vacuum mode.
+ * */
+ if (!IsParallelWorker())
+ {
+ pvs->pcxt->parallel_progress_callback(pvs->pcxt->parallel_progress_callback_arg);
+ }
+
/*
* Copy the index bulk-deletion result returned from ambulkdelete and
* amvacuumcleanup to the DSM segment if it's the first cycle because they
@@ -1071,3 +1089,27 @@ parallel_vacuum_error_callback(void *arg)
return;
}
}
+
+/*
+ * Callback to report index vacuum progress.
+ * Vacuum Progress should only be reported
+ * to the leader process.
+ */
+static void
+parallel_vacuum_progress_callback(void *arg)
+{
+ ParallelVacuumState *pvs = (ParallelVacuumState *)arg;
+ int indexes_completed = 0;
+
+ Assert(!IsParallelWorker());
+
+ for (int i = 0; i < pvs->nindexes; i++)
+ {
+ PVIndStats *indstats = &(pvs->indstats[i]);
+
+ if (indstats->status == PARALLEL_INDVAC_STATUS_COMPLETED)
+ indexes_completed++;
+ }
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed);
+}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..0d1724ee75 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_INDEXES_TOTAL 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 21effe8315..53ff7151c8 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2008,7 +2008,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_completed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT s.stats_reset,
--
2.32.0
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-05-02 03:30 ` Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-05-02 03:30 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Thu, Apr 14, 2022 at 10:32 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> Taking the above feedback, I modified the patches
> and I believe this approach should be acceptable.
>
> For now, I also reduced the scope to only exposing
> Indexes_total and indexes_completed in
> pg_stat_progress_vacuum. I will create a new CF entry
> for the new view pg_stat_progress_vacuum_index.
>
> V10-0001: This patch adds a callback to ParallelContext
> that could be called by the leader in vacuumparallel.c
> and more importantly while the leader is waiting
> for the parallel workers to complete inside.
Thank you for updating the patch! The new design looks much better to me.
typedef struct ParallelWorkerInfo
@@ -46,6 +49,8 @@ typedef struct ParallelContext
ParallelWorkerInfo *worker;
int nknown_attached_workers;
bool *known_attached_workers;
+ ParallelProgressCallback parallel_progress_callback;
+ void *parallel_progress_callback_arg;
} ParallelContext;
I think we can pass the progress update function to
WaitForParallelWorkersToFinish(), which seems simpler. And we can call
the function after updating the index status to
PARALLEL_INDVAC_STATUS_COMPLETED.
BTW, currently we don't need a lock for touching index status since
each worker touches different indexes. But after this patch, the
leader will touch all index status, do we need a lock for that?
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-05-05 19:26 ` Imseih (AWS), Sami <[email protected]>
2022-05-26 13:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-05-05 19:26 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Thank you for the feedback!
> I think we can pass the progress update function to
> WaitForParallelWorkersToFinish(), which seems simpler. And we can call
Directly passing the callback to WaitForParallelWorkersToFinish
will require us to modify the function signature.
To me, it seemed simpler and touches less code to have
the caller set the callback in the ParallelContext.
> the function after updating the index status to
> PARALLEL_INDVAC_STATUS_COMPLETED.
I also like this better. Will make the change.
> BTW, currently we don't need a lock for touching index status since
> each worker touches different indexes. But after this patch, the
> leader will touch all index status, do we need a lock for that?
I do not think locking is needed here. The leader and workers
will continue to touch different indexes to update the status.
However, if the process is a leader, it will call the function
which will go through indstats and count how many
Indexes have a status of PARALLEL_INDVAC_STATUS_COMPLETED.
This value is then reported to the leaders backend only.
Regards,
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-05-26 13:41 ` Imseih (AWS), Sami <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-05-26 13:41 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> the function after updating the index status to
> PARALLEL_INDVAC_STATUS_COMPLETED.
> I also like this better. Will make the change.
I updated the patch. The progress function is called after
updating index status to PARALLEL_INDVAC_STATUS_COMPLETED.
I believe all comments have been addressed at this point.
Regards,
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v11-0001-Add-progress-reporting-callback-to-ParallelConte.patch (2.6K, ../../[email protected]/2-v11-0001-Add-progress-reporting-callback-to-ParallelConte.patch)
download | inline diff:
From 19b1a9b4814fdf411a34a0ca86ce224079c857cc Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Thu, 26 May 2022 07:45:26 -0500
Subject: [PATCH v11 1/2] Add progress reporting callback to ParallelContext
The purpose of supporting a progress reporting
callback in ParallelContext is to allow for the
leader process to report progress while waiting
for workers to complete.
The first use-case for this is to report index
progress in pg_stat_progress_vacuum.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
src/backend/access/transam/parallel.c | 16 ++++++++++++++++
src/include/access/parallel.h | 5 +++++
2 files changed, 21 insertions(+)
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index df0cd77..bfe3275 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -774,6 +774,22 @@ WaitForParallelWorkersToFinish(ParallelContext *pcxt)
*/
CHECK_FOR_INTERRUPTS();
+ /*
+ * We call the parallel progress callback while
+ * waiting for the parallel workers to finish.
+ * This is to ensure that the leader keeps
+ * updating progress when waiting for
+ * parallel workers to finish.
+ *
+ * We must ensure that pcxt->parallel_progress_callback
+ * is set before calling as not all parallel
+ * operations will set a callback.
+ */
+ if (pcxt->parallel_progress_callback)
+ {
+ pcxt->parallel_progress_callback(pcxt->parallel_progress_callback_arg);
+ }
+
for (i = 0; i < pcxt->nworkers_launched; ++i)
{
/*
diff --git a/src/include/access/parallel.h b/src/include/access/parallel.h
index 983841d..53a3d13 100644
--- a/src/include/access/parallel.h
+++ b/src/include/access/parallel.h
@@ -20,6 +20,9 @@
#include "storage/shm_mq.h"
#include "storage/shm_toc.h"
+/* progress callback definition */
+typedef void (*ParallelProgressCallback) (void *parallel_progress_callback_state);
+
typedef void (*parallel_worker_main_type) (dsm_segment *seg, shm_toc *toc);
typedef struct ParallelWorkerInfo
@@ -46,6 +49,8 @@ typedef struct ParallelContext
ParallelWorkerInfo *worker;
int nknown_attached_workers;
bool *known_attached_workers;
+ ParallelProgressCallback parallel_progress_callback;
+ void *parallel_progress_callback_arg;
} ParallelContext;
typedef struct ParallelWorkerContext
--
2.32.1 (Apple Git-133)
[application/octet-stream] v11-0002-Show-progress-for-index-vacuums.patch (9.5K, ../../[email protected]/3-v11-0002-Show-progress-for-index-vacuums.patch)
download | inline diff:
From 5d0c7f7f761316d1089b010bc750cf36803b53e3 Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Thu, 26 May 2022 08:02:11 -0500
Subject: [PATCH v11 2/2] Show progress for index vacuums
Add 2 new columns to pg_stat_progress_vacuum. The columns are
indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so
far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 25 ++++++++++++++++++
src/backend/access/heap/vacuumlazy.c | 19 ++++++++++++++
src/backend/catalog/system_views.sql | 3 ++-
src/backend/commands/vacuumparallel.c | 37 +++++++++++++++++++++++++++
src/include/commands/progress.h | 2 ++
src/test/regress/expected/rules.out | 4 ++-
6 files changed, 88 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 56d9b37..0599384 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6396,6 +6396,31 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes to be processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when there are no indexes to process
+ or when failsafe is triggered during the vacuum.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_completed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ The number of indexes already processed in the
+ <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase. It is set to
+ <literal>0</literal> when vacuum is not in any of these phases.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index b802ed2..f01341e 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -418,6 +418,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+ /* Report the number of indexes to vacuum/cleanup */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2327,6 +2330,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /* Report the number of indexes vacuumed */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
if (lazy_check_wraparound_failsafe(vacrel))
{
@@ -2361,6 +2366,10 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
vacrel->dead_items->num_items == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
+ /* Report that we're done vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
/*
* Increase and report the number of index scans.
*
@@ -2625,6 +2634,10 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ /* Report that we're no longer vacuuming indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
+
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
get_database_name(MyDatabaseId),
@@ -2671,6 +2684,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+ /* Report the number of indexes cleaned */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, idx + 1);
}
}
else
@@ -2680,6 +2695,10 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->num_index_scans,
estimated_count);
}
+
+ /* Report that we're done cleaning up indexes */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, 0);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index fedaed5..0c8261b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1163,7 +1163,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_completed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 1753da6..ad61a54 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -30,6 +30,7 @@
#include "access/table.h"
#include "access/xact.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -213,6 +214,7 @@ static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation
static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_index_scans,
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
+static void parallel_vacuum_progress_callback(void *arg);
/*
* Try to enter parallel mode and create a parallel context. Then initialize
@@ -288,6 +290,10 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
+ /* Setup the Parallel Progress Callback */
+ pvs->pcxt->parallel_progress_callback = parallel_vacuum_progress_callback;
+ pvs->pcxt->parallel_progress_callback_arg = pvs;
+
/*
* Estimate space for BufferUsage and WalUsage --
* PARALLEL_VACUUM_KEY_BUFFER_USAGE and PARALLEL_VACUUM_KEY_WAL_USAGE.
@@ -885,6 +891,13 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
/* Reset error traceback information */
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
+
+ /*
+ * If we are the leader, update index vacuum progress.
+ */
+ if (!IsParallelWorker())
+ pvs->pcxt->parallel_progress_callback(pvs->pcxt->parallel_progress_callback_arg);
+
pfree(pvs->indname);
pvs->indname = NULL;
}
@@ -1071,3 +1084,27 @@ parallel_vacuum_error_callback(void *arg)
return;
}
}
+
+/*
+ * Callback to report index vacuum progress.
+ * Vacuum Progress should only be reported
+ * to the leader process.
+ */
+static void
+parallel_vacuum_progress_callback(void *arg)
+{
+ ParallelVacuumState *pvs = (ParallelVacuumState *)arg;
+ int indexes_completed = 0;
+
+ Assert(!IsParallelWorker());
+
+ for (int i = 0; i < pvs->nindexes; i++)
+ {
+ PVIndStats *indstats = &(pvs->indstats[i]);
+
+ if (indstats->status == PARALLEL_INDVAC_STATUS_COMPLETED)
+ indexes_completed++;
+ }
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_COMPLETED, indexes_completed);
+}
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938c..0d1724e 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_INDEXES_TOTAL 7
+#define PROGRESS_VACUUM_INDEXES_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index fc3cde3..9c3442f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2017,7 +2017,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_completed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT s.stats_reset,
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-05-26 15:43 ` Masahiko Sawada <[email protected]>
2022-05-27 01:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
1 sibling, 2 replies; 74+ messages in thread
From: Masahiko Sawada @ 2022-05-26 15:43 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Fri, May 6, 2022 at 4:26 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> Thank you for the feedback!
>
> > I think we can pass the progress update function to
> > WaitForParallelWorkersToFinish(), which seems simpler. And we can call
>
> Directly passing the callback to WaitForParallelWorkersToFinish
> will require us to modify the function signature.
>
> To me, it seemed simpler and touches less code to have
> the caller set the callback in the ParallelContext.
Okay, but if we do that, I think we should add comments about when
it's used. The callback is used only when
WaitForParallelWorkersToFinish(), but not when
WaitForParallelWorkersToExit().
Another idea I came up with is that we can wait for all index vacuums
to finish while checking and updating the progress information, and
then calls WaitForParallelWorkersToFinish after confirming all index
status became COMPLETED. That way, we don’t need to change the
parallel query infrastructure. What do you think?
>
> > the function after updating the index status to
> > PARALLEL_INDVAC_STATUS_COMPLETED.
>
> I also like this better. Will make the change.
>
> > BTW, currently we don't need a lock for touching index status since
> > each worker touches different indexes. But after this patch, the
> > leader will touch all index status, do we need a lock for that?
>
> I do not think locking is needed here. The leader and workers
> will continue to touch different indexes to update the status.
>
> However, if the process is a leader, it will call the function
> which will go through indstats and count how many
> Indexes have a status of PARALLEL_INDVAC_STATUS_COMPLETED.
> This value is then reported to the leaders backend only.
I was concerned that the leader process could report the wrong
progress if updating and checking index status happen concurrently.
But I think it should be fine since we can read PVIndVacStatus
atomically.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-05-27 01:52 ` Imseih (AWS), Sami <[email protected]>
2022-06-03 05:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-05-27 01:52 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> Another idea I came up with is that we can wait for all index vacuums
> to finish while checking and updating the progress information, and
> then calls WaitForParallelWorkersToFinish after confirming all index
> status became COMPLETED. That way, we don’t need to change the
> parallel query infrastructure. What do you think?
Thinking about this a bit more, the idea of using
WaitForParallelWorkersToFinish
Will not work if you have a leader worker that is
stuck on a large index. The progress will not be updated
until the leader completes. Even if the parallel workers
finish.
What are your thought about piggybacking on the
vacuum_delay_point to update progress. The leader can
perhaps keep a counter to update progress every few thousand
calls to vacuum_delay_point.
This goes back to your original idea to keep updating progress
while scanning the indexes.
/*
* vacuum_delay_point --- check for interrupts and cost-based delay.
*
* This should be called in each major loop of VACUUM processing,
* typically once per page processed.
*/
void
vacuum_delay_point(void)
{
---
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-27 01:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-06-03 05:39 ` Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Masahiko Sawada @ 2022-06-03 05:39 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Fri, May 27, 2022 at 10:52 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> > Another idea I came up with is that we can wait for all index vacuums
> > to finish while checking and updating the progress information, and
> > then calls WaitForParallelWorkersToFinish after confirming all index
> > status became COMPLETED. That way, we don’t need to change the
> > parallel query infrastructure. What do you think?
>
> Thinking about this a bit more, the idea of using
> WaitForParallelWorkersToFinish
> Will not work if you have a leader worker that is
> stuck on a large index. The progress will not be updated
> until the leader completes. Even if the parallel workers
> finish.
Right.
>
> What are your thought about piggybacking on the
> vacuum_delay_point to update progress. The leader can
> perhaps keep a counter to update progress every few thousand
> calls to vacuum_delay_point.
>
> This goes back to your original idea to keep updating progress
> while scanning the indexes.
I think we can have the leader process wait for all index statuses to
become COMPLETED before WaitForParallelWorkersToFinish(). While
waiting for it, the leader can update its progress information. After
the leader confirmed all index statuses became COMPLETED, it can wait
for the workers to finish by WaitForParallelWorkersToFinish().
Regarding waiting in vacuum_delay_point, it might be a side effect as
it’s called every page and used not only by vacuum such as analyze,
but it seems to be worth trying.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-06-06 14:41 ` Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Robert Haas @ 2022-06-06 14:41 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Imseih (AWS), Sami <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Thu, May 26, 2022 at 11:43 AM Masahiko Sawada <[email protected]> wrote:
> Another idea I came up with is that we can wait for all index vacuums
> to finish while checking and updating the progress information, and
> then calls WaitForParallelWorkersToFinish after confirming all index
> status became COMPLETED. That way, we don’t need to change the
> parallel query infrastructure. What do you think?
+1 from me. It doesn't seem to me that we should need to add something
like parallel_vacuum_progress_callback in order to solve this problem,
because the parallel index vacuum code could just do the waiting
itself, as you propose here.
The question Sami asks him his reply is a good one, though -- who is
to say that the leader only needs to update progress at the end, once
it's finished the index it's handling locally? There will need to be a
callback system of some kind to allow the leader to update progress as
other workers finish, even if the leader is still working. I am not
too sure that the idea of using the vacuum delay points is the best
plan. I think we should try to avoid piggybacking on such general
infrastructure if we can, and instead look for a way to tie this to
something that is specific to parallel vacuum. However, I haven't
studied the problem so I'm not sure whether there's a reasonable way
to do that.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
@ 2022-06-20 06:35 ` Masahiko Sawada <[email protected]>
2022-08-02 18:06 ` Re: Add index scan progress to pg_stat_progress_vacuum Jacob Champion <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 2 replies; 74+ messages in thread
From: Masahiko Sawada @ 2022-06-20 06:35 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Imseih (AWS), Sami <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Mon, Jun 6, 2022 at 11:42 PM Robert Haas <[email protected]> wrote:
>
> On Thu, May 26, 2022 at 11:43 AM Masahiko Sawada <[email protected]> wrote:
> > Another idea I came up with is that we can wait for all index vacuums
> > to finish while checking and updating the progress information, and
> > then calls WaitForParallelWorkersToFinish after confirming all index
> > status became COMPLETED. That way, we don’t need to change the
> > parallel query infrastructure. What do you think?
>
> +1 from me. It doesn't seem to me that we should need to add something
> like parallel_vacuum_progress_callback in order to solve this problem,
> because the parallel index vacuum code could just do the waiting
> itself, as you propose here.
>
> The question Sami asks him his reply is a good one, though -- who is
> to say that the leader only needs to update progress at the end, once
> it's finished the index it's handling locally? There will need to be a
> callback system of some kind to allow the leader to update progress as
> other workers finish, even if the leader is still working. I am not
> too sure that the idea of using the vacuum delay points is the best
> plan. I think we should try to avoid piggybacking on such general
> infrastructure if we can, and instead look for a way to tie this to
> something that is specific to parallel vacuum. However, I haven't
> studied the problem so I'm not sure whether there's a reasonable way
> to do that.
One idea would be to add a flag, say report_parallel_vacuum_progress,
to IndexVacuumInfo struct and expect index AM to check and update the
parallel index vacuum progress, say every 1GB blocks processed. The
flag is true only when the leader process is vacuuming an index.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-08-02 18:06 ` Jacob Champion <[email protected]>
1 sibling, 0 replies; 74+ messages in thread
From: Jacob Champion @ 2022-08-02 18:06 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; Robert Haas <[email protected]>; +Cc: Imseih (AWS), Sami <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
This entry has been waiting on author input for a while (our current
threshold is roughly two weeks), so I've marked it Returned with
Feedback.
Once you think the patchset is ready for review again, you (or any
interested party) can resurrect the patch entry by visiting
https://commitfest.postgresql.org/38/3617/
and changing the status to "Needs Review", and then changing the
status again to "Move to next CF". (Don't forget the second step;
hopefully we will have streamlined this in the near future!)
Thanks,
--Jacob
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-10-11 13:50 ` Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-10-11 13:50 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; Robert Haas <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> One idea would be to add a flag, say report_parallel_vacuum_progress,
> to IndexVacuumInfo struct and expect index AM to check and update the
> parallel index vacuum progress, say every 1GB blocks processed. The
> flag is true only when the leader process is vacuuming an index.
> Regards,
Sorry for the long delay on this. I have taken the approach as suggested
by Sawada-san and Robert and attached is v12.
1. The patch introduces a new counter in the same shared memory already
used by the parallel leader and workers to keep track of the number
of indexes completed. This way there is no reason to loop through
the index status every time we want to get the status of indexes completed.
2. A new function in vacuumparallel.c will be used to update
the progress of indexes completed by reading from the
counter created in point #1.
3. The function is called during the vacuum_delay_point as a
matter of convenience, since this is called in all major vacuum
loops. The function will only do something if the caller
sets a boolean to report progress. Doing so will also ensure
progress is being reported in case the parallel workers completed
before the leader.
4. Rather than adding any complexity to WaitForParallelWorkersToFinish
and introducing a new callback, vacuumparallel.c will wait until
the number of vacuum workers is 0 and then call
WaitForParallelWorkersToFinish as it does currently.
5. Went back to the idea of adding a new view called pg_stat_progress_vacuum_index
which is accomplished by adding a new type called VACUUM_PARALLEL in progress.h
Thanks,
Sami Imseih
Amazon Web Servies (AWS)
FYI: the above message was originally sent yesterday but
was created under a separate thread. Please ignore this
thread[1]
[1]: https://www.postgresql.org/message-id/flat/4CD97E17-B9E4-421E-9A53-4317C90EFF35%40amazon.com
Attachments:
[application/octet-stream] v12-0001--Show-progress-for-index-vacuums.patch (23.0K, ../../[email protected]/2-v12-0001--Show-progress-for-index-vacuums.patch)
download | inline diff:
From fd394f0bf01406f850206a6c4a81ff187a685a69 Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Mon, 10 Oct 2022 11:22:25 -0500
Subject: [PATCH v12 1/1] Add 2 new columns to pg_stat_progress_vacuum. The
columns are indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so far.
Also, introduce a new view called pg_stat_progress_vacuum_index that
exposes the current index being vacuumed.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
doc/src/sgml/monitoring.sgml | 92 ++++++++++++++++++++++
doc/src/sgml/ref/vacuum.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 44 ++++++++++-
src/backend/catalog/system_views.sql | 20 ++++-
src/backend/commands/vacuum.c | 6 ++
src/backend/commands/vacuumparallel.c | 106 +++++++++++++++++++++++++-
src/backend/utils/adt/pgstatfuncs.c | 2 +
src/include/commands/progress.h | 4 +
src/include/commands/vacuum.h | 2 +
src/include/utils/backend_progress.h | 7 +-
src/test/regress/expected/rules.out | 17 ++++-
11 files changed, 301 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 342b20ebeb..473c76e6e8 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -392,6 +392,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_progress_vacuum_index</structname><indexterm><primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+ </entry>
+ <entry>One row for each backend (including autovacuum worker processes) performing the <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase of a <command>VACUUM</command>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_cluster</structname><indexterm><primary>pg_stat_progress_cluster</primary></indexterm></entry>
<entry>One row for each backend running
@@ -6414,6 +6423,89 @@ FROM pg_stat_get_backend_idset() AS backendid;
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes that wil be vacuumed. This value will be
+ <literal>0</literal> if there are no indexes to vacuum or
+ vacuum failsafe is triggered. See <xref linkend="guc-vacuum-failsafe-age"/>
+ for more on vacuum failsafe.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_completed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes vacuumed in the current vacuum cycle.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <indexterm>
+ <primary>pg_stat_progress_vacuum_index</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>VACUUM</command> is running, the
+ <structname>pg_stat_progress_vacuum_index</structname> view will contain
+ one row for each backend (including autovacuum worker processes) performing
+ the <literal>vacuuming indexes</literal> or <literal>cleaning up indexes</literal>
+ phase of a <command>VACUUM</command>.
+ </para>
+
+ <table id="pg-stat-progress-vacuum-view_index" xreflabel="pg_stat_progress_vacuum_index">
+ <title><structname>pg_stat_progress_vacuum_index</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>leader_pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of the leader backend in a parallel <command>VACUUM</command>. This value
+ will match the <structfield>pid</structfield> value whenever the leader
+ is processing an index or the <command>VACUUM</command> is not using parallel.
+ This field can be joined to <structfield>pid</structfield>
+ of <structfield>pg_stat_progress_vacuum</structfield> to get more details about
+ the <command>VACUUM</command>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexrelid</structfield> <type>integer</type>
+ </para>
+ <para>
+ OID of the index being processed in the current vacuum phase.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml
index c582021d29..0c08d9ac6d 100644
--- a/doc/src/sgml/ref/vacuum.sgml
+++ b/doc/src/sgml/ref/vacuum.sgml
@@ -411,7 +411,13 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
<para>
Each backend running <command>VACUUM</command> without the
<literal>FULL</literal> option will report its progress in the
- <structname>pg_stat_progress_vacuum</structname> view. Backends running
+ <structname>pg_stat_progress_vacuum</structname> view.
+ <structname>pg_stat_progress_vacuum</structname> view. Whenever a
+ <command>VACUUM</command> is in the <literal>vacuuming indexes</literal>
+ or <literal>cleaning up indexes</literal> phase,
+ the current index being processed is reported in
+ <structname>pg_stat_progress_vacuum_index</structname>.
+ <command>VACUUM FULL</command> will report their progress in the Backends running
<command>VACUUM FULL</command> will instead report their progress in the
<structname>pg_stat_progress_cluster</structname> view. See
<xref linkend="vacuum-progress-reporting"/> and
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index dfbe37472f..b057d18dd9 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -350,8 +350,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
}
+ /* start the vacuum progress command and report the leader pid. */
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, MyProcPid);
/*
* Get OldestXmin cutoff, which is used to determine which deleted tuples
@@ -420,6 +422,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+
+ /* report number of indexes to vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2337,10 +2343,21 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* report the index relid being vacuumed */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /*
+ * Done vacuuming an index.
+ * Increment the indexes completed and reset the index relid to 0
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2384,6 +2401,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_VACUUMS,
vacrel->num_index_scans);
+ /*
+ * Reset the indexes completed at this point.
+ * If we end up in another index vacuum cycle, we will
+ * start counting from the start.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
+
return allindexes;
}
@@ -2633,10 +2657,17 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
{
vacrel->failsafe_active = true;
- /* Disable index vacuuming, index cleanup, and heap rel truncation */
+ /*
+ * Disable index vacuuming, index cleanup, and heap rel truncation
+ *
+ * Also, report to progress.h that we are no longer tracking
+ * index vacuum/cleanup.
+ */
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2681,9 +2712,20 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /*+ report the index relid being cleaned */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /*
+ * Done cleaning an index.
+ * Increment the indexes completed and reset the index relid to 0
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
}
}
else
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 55f7ec79e0..5b5e7b4080 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1162,10 +1162,28 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param10 AS indexes_total, S.param11 AS indexes_completed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_progress_vacuum_index AS
+ SELECT
+ S.pid AS pid,
+ S.param9 AS leader_pid,
+ S.param8 AS indexrelid
+ FROM pg_stat_get_progress_info('VACUUM') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 in (2, 4) AND S.param8 > 0
+ UNION ALL
+ SELECT
+ S.pid AS pid,
+ S.param9 AS leader_pid,
+ S.param8 AS indexrelid
+ FROM pg_stat_get_progress_info('VACUUM_PARALLEL') AS S
+ LEFT JOIN pg_database D ON S.datid = D.oid
+ WHERE S.param1 in (2, 4) AND S.param8 > 0;
+
CREATE VIEW pg_stat_progress_cluster AS
SELECT
S.pid AS pid,
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 7ccde07de9..19a2692704 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -2173,12 +2173,18 @@ vac_close_indexes(int nindexes, Relation *Irel, LOCKMODE lockmode)
*
* This should be called in each major loop of VACUUM processing,
* typically once per page processed.
+ *
+ * NOTE: For convenience, parallel_vacuum_progress_report, is called
+ * here so the leader can report the number of indexes vacuumed in
+ * while inside all the major VACUUM loops.
*/
void
vacuum_delay_point(void)
{
double msec = 0;
+ parallel_vacuum_progress_report();
+
/* Always check for interrupts */
CHECK_FOR_INTERRUPTS();
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index f26d796e52..65a71cbac9 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -30,6 +30,7 @@
#include "access/table.h"
#include "access/xact.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -103,6 +104,20 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /*
+ * Counter for vacuuming and cleanup progress reporting.
+ * This value is used to report index vacuum/cleanup progress
+ * in parallel_vacuum_progress_report. We keep this
+ * counter to avoid having to loop through
+ * ParallelVacuumState->indstats to determine the number
+ * of indexes completed.
+ */
+ pg_atomic_uint32 idx_completed_progress;
+
+ /* track the leader pid of a parallel vacuum */
+ int leader_pid;
+
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -214,6 +229,9 @@ static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_inde
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
+static pg_atomic_uint32 *index_vacuum_completed = NULL;
+static bool report_parallel_vacuum_progress = false;
+
/*
* Try to enter parallel mode and create a parallel context. Then initialize
* shared memory state.
@@ -364,6 +382,9 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
pg_atomic_init_u32(&(shared->idx), 0);
+ pg_atomic_init_u32(&(shared->idx_completed_progress), 0);
+
+ shared->leader_pid = MyProcPid;
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
@@ -618,8 +639,9 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
vacuum));
}
- /* Reset the parallel index processing counter */
+ /* Reset the parallel index processing counter ( index proress counter also ) */
pg_atomic_write_u32(&(pvs->shared->idx), 0);
+ pg_atomic_write_u32(&(pvs->shared->idx_completed_progress), 0);
/* Setup the shared cost-based vacuum delay and launch workers */
if (nworkers > 0)
@@ -657,6 +679,13 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
/* Enable shared cost balance for leader backend */
VacuumSharedCostBalance = &(pvs->shared->cost_balance);
VacuumActiveNWorkers = &(pvs->shared->active_nworkers);
+
+ /*
+ * If we are launching a parallel vacuum/cleanup,
+ * setup the tracking.
+ */
+ report_parallel_vacuum_progress = true;
+ index_vacuum_completed = &(pvs->shared->idx_completed_progress);
}
if (vacuum)
@@ -682,13 +711,36 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
*/
parallel_vacuum_process_safe_indexes(pvs);
+ /*
+ * In case the leader completes vacuuming all
+ * its indexes before the parallel workers do,
+ * it can spin here waiting for the number of
+ * active workers to complete while reporting
+ * the progress of indexes vacuumed.
+ *
+ */
+ if (VacuumActiveNWorkers)
+ {
+ while (pg_atomic_read_u32(VacuumActiveNWorkers) > 0)
+ {
+ parallel_vacuum_progress_report();
+ }
+ }
+
/*
* Next, accumulate buffer and WAL usage. (This must wait for the workers
* to finish, or we might get incomplete data.)
*/
if (nworkers > 0)
{
- /* Wait for all vacuum workers to finish */
+ /*
+ * Wait for all vacuum workers to finish
+ *
+ * We must do this even if we know we don't
+ * have anymore active workers. See the
+ * WaitForParallelWorkersToFinish commentary
+ * as to why.
+ */
WaitForParallelWorkersToFinish(pvs->pcxt);
for (int i = 0; i < pvs->pcxt->nworkers_launched; i++)
@@ -719,6 +771,14 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
VacuumSharedCostBalance = NULL;
VacuumActiveNWorkers = NULL;
}
+
+ /*
+ * Disabled index vacuum progress reporting.
+ * If we havee another index vacuum cycle, the
+ * progress reporting will be re-enabled then.
+ */
+ index_vacuum_completed = NULL;
+ report_parallel_vacuum_progress = false;
}
/*
@@ -823,6 +883,21 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
IndexBulkDeleteResult *istat = NULL;
IndexBulkDeleteResult *istat_res;
IndexVacuumInfo ivinfo;
+ Oid indrelid = RelationGetRelid(indrel);
+
+ /*
+ * If we are a parallel worker, start a PROGRESS_COMMAND_VACUUM_PARALLEL
+ * command for progress reporting, and set the leader pid.
+ */
+ if (IsParallelWorker())
+ {
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL,
+ pvs->shared->relid);
+ pgstat_progress_update_param(PROGRESS_VACUUM_LEADER_PID, pvs->shared->leader_pid);
+ }
+
+ /* report the index being vacuumed or cleaned up */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, indrelid);
/*
* Update the pointer to the corresponding bulk-deletion result if someone
@@ -846,9 +921,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
break;
default:
@@ -888,6 +965,17 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
pvs->indname = NULL;
+
+ /*
+ * Reset the index relid begin vacuumed and incremebts the
+ * index vacuum counter.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
+ pg_atomic_add_fetch_u32(&(pvs->shared->idx_completed_progress), 1);
+
+ /* if we are a parallel worker, end the command */
+ if (IsParallelWorker())
+ pgstat_progress_end_command();
}
/*
@@ -1072,3 +1160,17 @@ parallel_vacuum_error_callback(void *arg)
return;
}
}
+
+/*
+ * Read the number of indexes vacuumed from the shared counter
+ * and report it to progress.h
+ */
+void
+parallel_vacuum_progress_report(void)
+{
+ if (IsParallelWorker() || !report_parallel_vacuum_progress)
+ return;
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(index_vacuum_completed));
+}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index eadd8464ff..1584a9bd7d 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -484,6 +484,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
cmdtype = PROGRESS_COMMAND_BASEBACKUP;
else if (pg_strcasecmp(cmd, "COPY") == 0)
cmdtype = PROGRESS_COMMAND_COPY;
+ else if (pg_strcasecmp(cmd, "VACUUM_PARALLEL") == 0)
+ cmdtype = PROGRESS_COMMAND_VACUUM_PARALLEL;
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..410eec217b 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,10 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_INDRELID 7
+#define PROGRESS_VACUUM_LEADER_PID 8
+#define PROGRESS_VACUUM_INDEX_TOTAL 9
+#define PROGRESS_VACUUM_INDEX_COMPLETED 10
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5d816ba7f4..96901d7234 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -338,4 +338,6 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+extern void parallel_vacuum_progress_report(void);
+
#endif /* VACUUM_H */
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..74d9dfb4c7 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -17,6 +17,10 @@
/* ----------
* Command type for progress reporting purposes
+ *
+ * Note: PROGRESS_COMMAND_VACUUM_PARALLEL is not
+ * a command per se, but this type is used to track
+ * parallel vacuum workers progress.
* ----------
*/
typedef enum ProgressCommandType
@@ -27,7 +31,8 @@ typedef enum ProgressCommandType
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
PROGRESS_COMMAND_BASEBACKUP,
- PROGRESS_COMMAND_COPY
+ PROGRESS_COMMAND_COPY,
+ PROGRESS_COMMAND_VACUUM_PARALLEL
} ProgressCommandType;
#define PGSTAT_NUM_PROGRESS_PARAM 20
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 9dd137415e..d7d6148d52 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2013,9 +2013,24 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param10 AS indexes_total,
+ s.param11 AS indexes_completed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
+pg_stat_progress_vacuum_index| SELECT s.pid,
+ s.param9 AS leader_pid,
+ s.param8 AS indexrelid
+ FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE ((s.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint])) AND (s.param8 > 0))
+UNION ALL
+ SELECT s.pid,
+ s.param9 AS leader_pid,
+ s.param8 AS indexrelid
+ FROM (pg_stat_get_progress_info('VACUUM_PARALLEL'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
+ LEFT JOIN pg_database d ON ((s.datid = d.oid)))
+ WHERE ((s.param1 = ANY (ARRAY[(2)::bigint, (4)::bigint])) AND (s.param8 > 0));
pg_stat_recovery_prefetch| SELECT s.stats_reset,
s.prefetch,
s.hit,
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-10-12 07:15 ` Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-10-12 07:15 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Tue, Oct 11, 2022 at 10:50 PM Imseih (AWS), Sami <[email protected]> wrote:
>
> > One idea would be to add a flag, say report_parallel_vacuum_progress,
> > to IndexVacuumInfo struct and expect index AM to check and update the
> > parallel index vacuum progress, say every 1GB blocks processed. The
> > flag is true only when the leader process is vacuuming an index.
>
> > Regards,
>
> Sorry for the long delay on this. I have taken the approach as suggested
> by Sawada-san and Robert and attached is v12.
Thank you for updating the patch!
>
> 1. The patch introduces a new counter in the same shared memory already
> used by the parallel leader and workers to keep track of the number
> of indexes completed. This way there is no reason to loop through
> the index status every time we want to get the status of indexes completed.
While it seems to be a good idea to have the atomic counter for the
number of indexes completed, I think we should not use the global
variable referencing the counter as follow:
+static pg_atomic_uint32 *index_vacuum_completed = NULL;
:
+void
+parallel_vacuum_progress_report(void)
+{
+ if (IsParallelWorker() || !report_parallel_vacuum_progress)
+ return;
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(index_vacuum_completed));
+}
I think we can pass ParallelVacuumState (or PVIndStats) to the
reporting function so that it can check the counter and report the
progress.
> 2. A new function in vacuumparallel.c will be used to update
> the progress of indexes completed by reading from the
> counter created in point #1.
>
> 3. The function is called during the vacuum_delay_point as a
> matter of convenience, since this is called in all major vacuum
> loops. The function will only do something if the caller
> sets a boolean to report progress. Doing so will also ensure
> progress is being reported in case the parallel workers completed
> before the leader.
Robert pointed out:
---
I am not too sure that the idea of using the vacuum delay points is the best
plan. I think we should try to avoid piggybacking on such general
infrastructure if we can, and instead look for a way to tie this to
something that is specific to parallel vacuum.
---
I agree with this part.
Instead, I think we can add a boolean and the pointer for
ParallelVacuumState to IndexVacuumInfo. If the boolean is true, an
index AM can call the reporting function with the pointer to
ParallelVacuumState while scanning index blocks, for example, for
every 1GB. The boolean can be true only for the leader process.
>
> 4. Rather than adding any complexity to WaitForParallelWorkersToFinish
> and introducing a new callback, vacuumparallel.c will wait until
> the number of vacuum workers is 0 and then call
> WaitForParallelWorkersToFinish as it does currently.
Agreed, but with the following change, the leader process waits in a
busy loop, which should not be acceptable:
+ if (VacuumActiveNWorkers)
+ {
+ while (pg_atomic_read_u32(VacuumActiveNWorkers) > 0)
+ {
+ parallel_vacuum_progress_report();
+ }
+ }
+
Also, I think it's better to check whether idx_completed_progress
equals to the number indexes instead.
> 5. Went back to the idea of adding a new view called pg_stat_progress_vacuum_index
> which is accomplished by adding a new type called VACUUM_PARALLEL in progress.h
Probably, we can devide the patch into two patches. One for adding the
new statistics of the number of vacuumed indexes to
pg_stat_progress_vacuum and another one for adding new statistics view
pg_stat_progress_vacuum_index.
Regards,
--
Masahiko Sawada
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-10-14 20:05 ` Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-10-14 20:05 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Thank you for the feedback!
> While it seems to be a good idea to have the atomic counter for the
> number of indexes completed, I think we should not use the global
> variable referencing the counter as follow:
> +static pg_atomic_uint32 *index_vacuum_completed = NULL;
> :
> +void
> +parallel_vacuum_progress_report(void)
> +{
> + if (IsParallelWorker() || !report_parallel_vacuum_progress)
> + return;
> +
> + pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
> + pg_atomic_read_u32(index_vacuum_completed));
> +}
> I think we can pass ParallelVacuumState (or PVIndStats) to the
> reporting function so that it can check the counter and report the
> progress.
Yes, that makes sense.
> ---
> I am not too sure that the idea of using the vacuum delay points is the best
> plan. I think we should try to avoid piggybacking on such general
> infrastructure if we can, and instead look for a way to tie this to
> something that is specific to parallel vacuum.
> ---
Adding the call to vacuum_delay_point made sense since it's
called in all major vacuum scans. While it is also called
by analyze, it will only do anything if the caller sets a flag
to report_parallel_vacuum_progress.
> Instead, I think we can add a boolean and the pointer for
> ParallelVacuumState to IndexVacuumInfo. If the boolean is true, an
> index AM can call the reporting function with the pointer to
> ParallelVacuumState while scanning index blocks, for example, for
> every 1GB. The boolean can be true only for the leader process.
Will doing this every 1GB be necessary? Considering the function
will not do much more than update progress from the value
stored in index_vacuum_completed?
> Agreed, but with the following change, the leader process waits in a
> busy loop, which should not be acceptable:
Good point.
> Also, I think it's better to check whether idx_completed_progress
equals to the number indexes instead.
Good point
> 5. Went back to the idea of adding a new view called pg_stat_progress_vacuum_index
> which is accomplished by adding a new type called VACUUM_PARALLEL in progress.h
> Probably, we can devide the patch into two patches. One for adding the
Makes sense.
Thanks
Sami Imseih
Amazon Web Services (AWS)
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-11-02 16:52 ` Imseih (AWS), Sami <[email protected]>
2022-11-03 08:16 ` Re: Add index scan progress to pg_stat_progress_vacuum Ian Lawrence Barwick <[email protected]>
2022-11-04 13:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-08 07:49 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 3 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-11-02 16:52 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Attached is v13-0001--Show-progress-for-index-vacuums.patch which addresses
the latest comments. The main changes are:
1/ Call the parallel_vacuum_progress_report inside the AMs rather than vacuum_delay_point.
2/ A Boolean when set to True in vacuumparallel.c will be used to determine if calling
parallel_vacuum_progress_report is necessary.
3/ Removed global varilable from vacuumparallel.c
4/ Went back to calling parallel_vacuum_progress_report inside
WaitForParallelWorkersToFinish to cover the case when a
leader is waiting for parallel workers to finish.
5/ I did not see a need to only report progress after 1GB as it's a fairly cheap call to update
progress.
6/ v1-0001-Function-to-return-currently-vacuumed-or-cleaned-ind.patch is a separate patch
for exposing the index relid being vacuumed by a backend.
Thanks
Sami Imseih
Amazon Web Services (AWS)
Attachments:
[application/octet-stream] v13-0001--Show-progress-for-index-vacuums.patch (24.3K, ../../[email protected]/2-v13-0001--Show-progress-for-index-vacuums.patch)
download | inline diff:
From e080c28d09567cb3dd974c0855487c717dad0a57 Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Tue, 1 Nov 2022 11:50:14 -0500
Subject: [PATCH 1/1] Add 2 new columns to pg_stat_progress_vacuum. The columns
are indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
contrib/bloom/blvacuum.c | 6 +++
doc/src/sgml/monitoring.sgml | 21 +++++++++
src/backend/access/brin/brin.c | 9 ++--
src/backend/access/gin/ginvacuum.c | 9 ++++
src/backend/access/gist/gistvacuum.c | 5 +++
src/backend/access/hash/hash.c | 8 +++-
src/backend/access/hash/hashpage.c | 4 +-
src/backend/access/heap/vacuumlazy.c | 40 ++++++++++++++++-
src/backend/access/nbtree/nbtree.c | 3 ++
src/backend/access/spgist/spgvacuum.c | 3 ++
src/backend/access/transam/parallel.c | 5 +++
src/backend/catalog/index.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 63 ++++++++++++++++++++++++++-
src/include/access/genam.h | 3 ++
src/include/access/hash.h | 3 +-
src/include/access/parallel.h | 2 +
src/include/commands/progress.h | 2 +
src/include/commands/vacuum.h | 2 +
src/test/regress/expected/rules.out | 4 +-
20 files changed, 184 insertions(+), 12 deletions(-)
diff --git a/contrib/bloom/blvacuum.c b/contrib/bloom/blvacuum.c
index 91fae5b0c0..19ae819cfe 100644
--- a/contrib/bloom/blvacuum.c
+++ b/contrib/bloom/blvacuum.c
@@ -136,6 +136,9 @@ blbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
GenericXLogAbort(gxlogState);
}
UnlockReleaseBuffer(buffer);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/*
@@ -209,6 +212,9 @@ blvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
}
UnlockReleaseBuffer(buffer);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
IndexFreeSpaceMapVacuum(info->index);
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index e5d622d514..c61443f3d5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6455,6 +6455,27 @@ FROM pg_stat_get_backend_idset() AS backendid;
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes that wil be vacuumed. This value will be
+ <literal>0</literal> if there are no indexes to vacuum or
+ vacuum failsafe is triggered. See <xref linkend="guc-vacuum-failsafe-age"/>
+ for more on vacuum failsafe.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_completed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes vacuumed in the current vacuum cycle.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 20b7d65b94..bd96ecf41e 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -78,7 +78,7 @@ static void brinsummarize(Relation index, Relation heapRel, BlockNumber pageRang
static void form_and_insert_tuple(BrinBuildState *state);
static void union_tuples(BrinDesc *bdesc, BrinMemTuple *a,
BrinTuple *b);
-static void brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy);
+static void brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy, IndexVacuumInfo *info);
static bool add_values_to_range(Relation idxRel, BrinDesc *bdesc,
BrinMemTuple *dtup, Datum *values, bool *nulls);
static bool check_null_keys(BrinValues *bval, ScanKey *nullkeys, int nnullkeys);
@@ -952,7 +952,7 @@ brinvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
heapRel = table_open(IndexGetRelation(RelationGetRelid(info->index), false),
AccessShareLock);
- brin_vacuum_scan(info->index, info->strategy);
+ brin_vacuum_scan(info->index, info->strategy, info);
brinsummarize(info->index, heapRel, BRIN_ALL_BLOCKRANGES, false,
&stats->num_index_tuples, &stats->num_index_tuples);
@@ -1659,7 +1659,7 @@ union_tuples(BrinDesc *bdesc, BrinMemTuple *a, BrinTuple *b)
* and such.
*/
static void
-brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy)
+brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy, IndexVacuumInfo *info)
{
BlockNumber nblocks;
BlockNumber blkno;
@@ -1681,6 +1681,9 @@ brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy)
brin_page_cleanup(idxrel, buf);
ReleaseBuffer(buf);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/*
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..3d5e4600dc 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -633,6 +633,9 @@ ginbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
UnlockReleaseBuffer(buffer);
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/* right now we found leftmost page in entry's BTree */
@@ -677,6 +680,9 @@ ginbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
LockBuffer(buffer, GIN_EXCLUSIVE);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
MemoryContextDelete(gvs.tmpCxt);
@@ -775,6 +781,9 @@ ginvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
}
UnlockReleaseBuffer(buffer);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/* Update the metapage with accurate page and entry counts */
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 0aa6e58a62..da3a0540d0 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -223,7 +223,12 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
break;
/* Iterate over pages, then loop back to recheck length */
for (; blkno < num_pages; blkno++)
+ {
gistvacuumpage(&vstate, blkno, blkno);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
+ }
}
/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index c361509d68..14790adf61 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -550,7 +550,7 @@ loop_top:
cachedmetap->hashm_highmask,
cachedmetap->hashm_lowmask, &tuples_removed,
&num_index_tuples, split_cleanup,
- callback, callback_state);
+ callback, callback_state, info);
_hash_dropbuf(rel, bucket_buf);
@@ -686,7 +686,8 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
uint32 maxbucket, uint32 highmask, uint32 lowmask,
double *tuples_removed, double *num_index_tuples,
bool split_cleanup,
- IndexBulkDeleteCallback callback, void *callback_state)
+ IndexBulkDeleteCallback callback, void *callback_state,
+ IndexVacuumInfo *info)
{
BlockNumber blkno;
Buffer buf;
@@ -775,6 +776,9 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
if (num_index_tuples)
*num_index_tuples += 1;
}
+
+ if (info && info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/* retain the pin on primary bucket page till end of bucket scan */
diff --git a/src/backend/access/hash/hashpage.c b/src/backend/access/hash/hashpage.c
index d2edcd4617..e9ce1a1110 100644
--- a/src/backend/access/hash/hashpage.c
+++ b/src/backend/access/hash/hashpage.c
@@ -759,7 +759,7 @@ restart_expand:
hashbucketcleanup(rel, old_bucket, buf_oblkno, start_oblkno, NULL,
maxbucket, highmask, lowmask, NULL, NULL, true,
- NULL, NULL);
+ NULL, NULL, NULL);
_hash_dropbuf(rel, buf_oblkno);
@@ -1327,7 +1327,7 @@ _hash_splitbucket(Relation rel,
hashbucketcleanup(rel, obucket, bucket_obuf,
BufferGetBlockNumber(bucket_obuf), NULL,
maxbucket, highmask, lowmask, NULL, NULL, true,
- NULL, NULL);
+ NULL, NULL, NULL);
}
else
{
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index dfbe37472f..120a2b18d7 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -420,6 +420,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+
+ /* report number of indexes to vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2341,6 +2345,12 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /*
+ * Done vacuuming an index. Increment the indexes completed
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2384,6 +2394,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_VACUUMS,
vacrel->num_index_scans);
+ /*
+ * Reset the indexes completed at this point.
+ * If we end up in another index vacuum cycle, we will
+ * start counting from the start.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
+
return allindexes;
}
@@ -2633,10 +2650,17 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
{
vacrel->failsafe_active = true;
- /* Disable index vacuuming, index cleanup, and heap rel truncation */
+ /*
+ * Disable index vacuuming, index cleanup, and heap rel truncation
+ *
+ * Also, report to progress.h that we are no longer tracking
+ * index vacuum/cleanup.
+ */
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2684,6 +2708,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /*
+ * Done cleaning an index. Increment the indexes completed
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
}
}
else
@@ -2693,6 +2723,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->num_index_scans,
estimated_count);
}
+
+ /*
+ * Reset the indexes completed at this point.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
+
}
/*
@@ -2718,6 +2754,7 @@ lazy_vacuum_one_index(Relation indrel, IndexBulkDeleteResult *istat,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = true;
ivinfo.message_level = DEBUG2;
ivinfo.num_heap_tuples = reltuples;
@@ -2766,6 +2803,7 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = estimated_count;
ivinfo.message_level = DEBUG2;
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index b52eca8f38..9904a9eb0c 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -998,6 +998,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
if (info->report_progress)
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
scanblkno);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
}
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..b1ab491980 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -843,6 +843,9 @@ spgvacuumscan(spgBulkDeleteState *bds)
/* empty the pending-list after each page */
if (bds->pendingList != NULL)
spgprocesspending(bds);
+
+ if (bds->info->report_parallel_progress)
+ bds->info->parallel_progress_callback(bds->info->parallel_progress_arg);
}
}
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index ee0985c7ed..0b4bde8297 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -185,6 +185,8 @@ CreateParallelContext(const char *library_name, const char *function_name,
pcxt->library_name = pstrdup(library_name);
pcxt->function_name = pstrdup(function_name);
pcxt->error_context_stack = error_context_stack;
+ pcxt->parallel_progress_callback = NULL;
+ pcxt->parallel_progress_arg = NULL;
shm_toc_initialize_estimator(&pcxt->estimator);
dlist_push_head(&pcxt_list, &pcxt->node);
@@ -785,6 +787,9 @@ WaitForParallelWorkersToFinish(ParallelContext *pcxt)
*/
CHECK_FOR_INTERRUPTS();
+ if (pcxt->parallel_progress_callback)
+ pcxt->parallel_progress_callback(pcxt->parallel_progress_arg);
+
for (i = 0; i < pcxt->nworkers_launched; ++i)
{
/*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 61f1d3926a..11b32129a7 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -3348,6 +3348,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
ivinfo.index = indexRelation;
ivinfo.analyze_only = false;
ivinfo.report_progress = true;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = true;
ivinfo.message_level = DEBUG2;
ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2d8104b090..c37b20b91b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1165,7 +1165,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_completed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index f26d796e52..85e2de5d54 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -30,6 +30,7 @@
#include "access/table.h"
#include "access/xact.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -103,6 +104,17 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /*
+ * Counter for vacuuming and cleanup progress reporting.
+ * This value is used to report index vacuum/cleanup progress
+ * in parallel_vacuum_progress_report. We keep this
+ * counter to avoid having to loop through
+ * ParallelVacuumState->indstats to determine the number
+ * of indexes completed.
+ */
+ pg_atomic_uint32 idx_completed_progress;
+
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -273,6 +285,9 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
Assert(pcxt->nworkers > 0);
pvs->pcxt = pcxt;
+ pvs->pcxt->parallel_progress_callback = parallel_vacuum_progress_report;
+ pvs->pcxt->parallel_progress_arg = pvs;
+
/* Estimate size for index vacuum stats -- PARALLEL_VACUUM_KEY_INDEX_STATS */
est_indstats_len = mul_size(sizeof(PVIndStats), nindexes);
shm_toc_estimate_chunk(&pcxt->estimator, est_indstats_len);
@@ -364,6 +379,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
pg_atomic_init_u32(&(shared->idx), 0);
+ pg_atomic_init_u32(&(shared->idx_completed_progress), 0);
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
@@ -618,8 +634,9 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
vacuum));
}
- /* Reset the parallel index processing counter */
+ /* Reset the parallel index processing counter ( index progress counter also ) */
pg_atomic_write_u32(&(pvs->shared->idx), 0);
+ pg_atomic_write_u32(&(pvs->shared->idx_completed_progress), 0);
/* Setup the shared cost-based vacuum delay and launch workers */
if (nworkers > 0)
@@ -834,10 +851,13 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = true;
ivinfo.message_level = DEBUG2;
ivinfo.estimated_count = pvs->shared->estimated_count;
ivinfo.num_heap_tuples = pvs->shared->reltuples;
ivinfo.strategy = pvs->bstrategy;
+ ivinfo.parallel_progress_callback = parallel_vacuum_progress_report;
+ ivinfo.parallel_progress_arg = pvs;
/* Update error traceback information */
pvs->indname = pstrdup(RelationGetRelationName(indrel));
@@ -846,9 +866,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
break;
default:
@@ -888,6 +910,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
pvs->indname = NULL;
+
+ /*
+ * Update the number of indexes completed.
+ */
+ pg_atomic_add_fetch_u32(&(pvs->shared->idx_completed_progress), 1);
}
/*
@@ -1072,3 +1099,37 @@ parallel_vacuum_error_callback(void *arg)
return;
}
}
+
+/*
+ * Read the number of indexes vacuumed from the shared counter
+ * and report it to progress.h
+ *
+ * This function is only called by the leader process of
+ * a parallel vacuum.
+ *
+ * If report_parallel_progress is set to true,
+ * this function is called from major loops inside the
+ * ambulkdelete and amvacuumcleanup.
+ *
+ * This function is also called inside WaitForParallelWorkersToFinish
+ * when report_parallel_progress is set to true.
+ *
+ * The reason for calling this function in both vacuum AM's
+ * and WaitForParallelWorkersToFinish is to ensure that
+ * parallel vacuum progress is constantly being reported
+ * if the leader process is either waiting for parallel
+ * proceses to finish (WaitForParallelWorkersToFinish) or
+ * the leader process is the last to finish and still
+ * inside the vacuum AM's.
+ */
+void
+parallel_vacuum_progress_report(void *arg)
+{
+ ParallelVacuumState *pvs = (ParallelVacuumState *) arg;
+
+ if (IsParallelWorker())
+ return;
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(pvs->shared->idx_completed_progress)));
+}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index e1c4fdbd03..6d5fc189f4 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -46,10 +46,13 @@ typedef struct IndexVacuumInfo
Relation index; /* the index being vacuumed */
bool analyze_only; /* ANALYZE (without any actual vacuum) */
bool report_progress; /* emit progress.h status reports */
+ bool report_parallel_progress; /* emit progress.h status reports for parallel vacuum */
bool estimated_count; /* num_heap_tuples is an estimate */
int message_level; /* ereport level for progress messages */
double num_heap_tuples; /* tuples remaining in heap */
BufferAccessStrategy strategy; /* access strategy for reads */
+ void (*parallel_progress_callback)(void *arg);
+ void *parallel_progress_arg;
} IndexVacuumInfo;
/*
diff --git a/src/include/access/hash.h b/src/include/access/hash.h
index da372841c4..14fb75a4ad 100644
--- a/src/include/access/hash.h
+++ b/src/include/access/hash.h
@@ -480,6 +480,7 @@ extern void hashbucketcleanup(Relation rel, Bucket cur_bucket,
uint32 maxbucket, uint32 highmask, uint32 lowmask,
double *tuples_removed, double *num_index_tuples,
bool split_cleanup,
- IndexBulkDeleteCallback callback, void *callback_state);
+ IndexBulkDeleteCallback callback, void *callback_state,
+ IndexVacuumInfo *info);
#endif /* HASH_H */
diff --git a/src/include/access/parallel.h b/src/include/access/parallel.h
index 1ec8e33af4..77e9c0a517 100644
--- a/src/include/access/parallel.h
+++ b/src/include/access/parallel.h
@@ -46,6 +46,8 @@ typedef struct ParallelContext
ParallelWorkerInfo *worker;
int nknown_attached_workers;
bool *known_attached_workers;
+ void (*parallel_progress_callback)(void *arg);
+ void *parallel_progress_arg;
} ParallelContext;
typedef struct ParallelWorkerContext
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..0e97c6d4ef 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_INDEX_TOTAL 7
+#define PROGRESS_VACUUM_INDEX_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5d816ba7f4..9510d028d3 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -338,4 +338,6 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+extern void parallel_vacuum_progress_report(void *arg);
+
#endif /* VACUUM_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 624d0e5aae..e0bf81247f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2018,7 +2018,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_completed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT s.stats_reset,
--
2.32.1 (Apple Git-133)
[application/octet-stream] v1-0001-Function-to-return-currently-vacuumed-or-cleaned-ind.patch (10.2K, ../../[email protected]/3-v1-0001-Function-to-return-currently-vacuumed-or-cleaned-ind.patch)
download | inline diff:
From 03f716bb9b1a79884655d87a6b3986a622aa4285 Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Wed, 2 Nov 2022 10:36:10 -0500
Subject: [PATCH 1/1] Function to return currently vacuumed ( or cleaned )
index
pg_stat_get_vacuum_index takes a PID as an argument and
returns the index relid of the currently vacuumed index.
NULL is returned if the backend is not involved in an index vacuum.
---
doc/src/sgml/func.sgml | 13 +++++++
doc/src/sgml/monitoring.sgml | 8 +++--
src/backend/access/heap/vacuumlazy.c | 9 +++++
src/backend/commands/vacuumparallel.c | 23 ++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 52 +++++++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 4 +++
src/include/commands/progress.h | 1 +
src/include/utils/backend_progress.h | 7 +++-
8 files changed, 114 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 6e0425cb3d..39841354de 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -22670,6 +22670,19 @@ SELECT * FROM pg_ls_dir('.') WITH ORDINALITY AS t(ls,n);
parsing the text version.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_stat_get_vacuum_index</primary>
+ </indexterm>
+ <function>pg_stat_get_vacuum_index</function> ( <type>integer</type> )
+ <returnvalue>oid</returnvalue>
+ </para>
+ <para>
+ Returns the relid of an index current being vacuumed or cleaned by the backend.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index c61443f3d5..8d947d8c3a 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6517,7 +6517,9 @@ FROM pg_stat_get_backend_idset() AS backendid;
has been completely scanned. It may happen multiple times per vacuum
if <xref linkend="guc-maintenance-work-mem"/> (or, in the case of autovacuum,
<xref linkend="guc-autovacuum-work-mem"/> if set) is insufficient to store
- the number of dead tuples found.
+ the number of dead tuples found. While in this phase, the function
+ <function>pg_stat_get_vacuum_index()</function> will return the relid
+ of the index being vacuumed.
</entry>
</row>
<row>
@@ -6536,7 +6538,9 @@ FROM pg_stat_get_backend_idset() AS backendid;
<entry>
<command>VACUUM</command> is currently cleaning up indexes. This occurs after
the heap has been completely scanned and all vacuuming of the indexes
- and the heap has been completed.
+ and the heap has been completed. While in this phase, the function
+ <function>pg_stat_get_vacuum_index()</function> will return the relid
+ of the index being cleaned.
</entry>
</row>
<row>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 120a2b18d7..8cd8e34f88 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -350,6 +350,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
}
+ /* start the vacuum progress command and report to progress.h */
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
@@ -2341,6 +2342,9 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* report the index relid being vacuumed */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
@@ -2350,6 +2354,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
*/
pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
if (lazy_check_wraparound_failsafe(vacrel))
{
@@ -2705,6 +2710,9 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* report the index relid being cleaned */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
@@ -2714,6 +2722,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
*/
pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
}
}
else
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 85e2de5d54..f2f51e86eb 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -840,6 +840,20 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
IndexBulkDeleteResult *istat = NULL;
IndexBulkDeleteResult *istat_res;
IndexVacuumInfo ivinfo;
+ Oid indrelid = RelationGetRelid(indrel);
+
+ /*
+ * If we are a parallel worker, start a PROGRESS_COMMAND_VACUUM_PARALLEL
+ * command for progress reporting and report the index being vacuumed.
+ */
+ if (IsParallelWorker())
+ {
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL,
+ pvs->shared->relid);
+ }
+
+ /* report the index being vacuumed or cleaned up */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, indrelid);
/*
* Update the pointer to the corresponding bulk-deletion result if someone
@@ -911,10 +925,19 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pfree(pvs->indname);
pvs->indname = NULL;
+ /*
+ * Reset the index relid begin vacuumed/cleaned
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
+
/*
* Update the number of indexes completed.
*/
pg_atomic_add_fetch_u32(&(pvs->shared->idx_completed_progress), 1);
+
+ /* if we are a parallel worker, end the command */
+ if (IsParallelWorker())
+ pgstat_progress_end_command();
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 96bffc0f2a..b08c15e7aa 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -19,6 +19,7 @@
#include "access/xlogprefetcher.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/progress.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -2383,3 +2384,54 @@ pg_stat_have_stats(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(pgstat_have_entry(kind, dboid, objoid));
}
+
+/*
+ * For a given PID, return the relid of the index
+ * currently being vacuumed.
+ */
+Datum
+pg_stat_get_vacuum_index(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ int num_backends = pgstat_fetch_stat_numbackends();
+ int curr_backend;
+ Oid indrelid = 0;
+ ProgressCommandType cmdtype_vacuum = PROGRESS_COMMAND_VACUUM;
+ ProgressCommandType cmdtype_vacuum_parallel = PROGRESS_COMMAND_VACUUM_PARALLEL;
+
+ /* 1-based index */
+ for (curr_backend = 1; curr_backend <= num_backends; curr_backend++)
+ {
+ LocalPgBackendStatus *local_beentry;
+ PgBackendStatus *beentry;
+
+ local_beentry = pgstat_fetch_stat_local_beentry(curr_backend);
+ beentry = &local_beentry->backendStatus;
+
+ /*
+ * If a backend has a VACUUM or VACUUM_PARALLEL progress command type
+ * and the procpid matches the given PID, return the index relid id
+ * that is current being vacuumed. In some cases, the index relid could
+ * be a '0' if we got here at thhe end of a vacuum or when no indexes
+ * are being vacuumed. In this case return NULL.
+ */
+ if (
+ (beentry->st_progress_command == cmdtype_vacuum
+ || beentry->st_progress_command == cmdtype_vacuum_parallel) &&
+ beentry->st_procpid == pid
+ )
+ {
+ indrelid = Int64GetDatum(beentry->st_progress_param[PROGRESS_VACUUM_INDRELID]);
+
+ if (indrelid > 0)
+ PG_RETURN_OID(indrelid);
+ else
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * No matching PID performing a VACUUM operation, return NULL.
+ */
+ PG_RETURN_NULL();
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 20f5aa56ea..dd5a32172a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11827,5 +11827,9 @@
proname => 'brin_minmax_multi_summary_send', provolatile => 's',
prorettype => 'bytea', proargtypes => 'pg_brin_minmax_multi_summary',
prosrc => 'brin_minmax_multi_summary_send' },
+{ oid => '4642',
+ descr => 'get the relid of the index being vacuumed by a backend',
+ proname => 'pg_stat_get_vacuum_index', provolatile => 'v', prorettype => 'int8',
+ proargtypes => 'int4', prosrc => 'pg_stat_get_vacuum_index' },
]
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 0e97c6d4ef..bc2b7d1e95 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -27,6 +27,7 @@
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
#define PROGRESS_VACUUM_INDEX_TOTAL 7
#define PROGRESS_VACUUM_INDEX_COMPLETED 8
+#define PROGRESS_VACUUM_INDRELID 9
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..74d9dfb4c7 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -17,6 +17,10 @@
/* ----------
* Command type for progress reporting purposes
+ *
+ * Note: PROGRESS_COMMAND_VACUUM_PARALLEL is not
+ * a command per se, but this type is used to track
+ * parallel vacuum workers progress.
* ----------
*/
typedef enum ProgressCommandType
@@ -27,7 +31,8 @@ typedef enum ProgressCommandType
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
PROGRESS_COMMAND_BASEBACKUP,
- PROGRESS_COMMAND_COPY
+ PROGRESS_COMMAND_COPY,
+ PROGRESS_COMMAND_VACUUM_PARALLEL
} ProgressCommandType;
#define PGSTAT_NUM_PROGRESS_PARAM 20
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-11-03 08:16 ` Ian Lawrence Barwick <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Ian Lawrence Barwick @ 2022-11-03 08:16 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
2022年11月3日(木) 1:52 Imseih (AWS), Sami <[email protected]>:
>
> Attached is v13-0001--Show-progress-for-index-vacuums.patch which addresses
> the latest comments. The main changes are:
>
> 1/ Call the parallel_vacuum_progress_report inside the AMs rather than vacuum_delay_point.
>
> 2/ A Boolean when set to True in vacuumparallel.c will be used to determine if calling
> parallel_vacuum_progress_report is necessary.
>
> 3/ Removed global varilable from vacuumparallel.c
>
> 4/ Went back to calling parallel_vacuum_progress_report inside
> WaitForParallelWorkersToFinish to cover the case when a
> leader is waiting for parallel workers to finish.
>
> 5/ I did not see a need to only report progress after 1GB as it's a fairly cheap call to update
> progress.
>
> 6/ v1-0001-Function-to-return-currently-vacuumed-or-cleaned-ind.patch is a separate patch
> for exposing the index relid being vacuumed by a backend.
This entry was marked "Needs review" in the CommitFest app but cfbot
reports the patch [1] no longer applies.
[1] this patch:
v1-0001-Function-to-return-currently-vacuumed-or-cleaned-ind.patch
We've marked it as "Waiting on Author". As CommitFest 2022-11 is
currently underway, this would be an excellent time update the patch.
Once you think the patchset is ready for review again, you (or any
interested party) can move the patch entry forward by visiting
https://commitfest.postgresql.org/40/3617/
and changing the status to "Needs review".
Thanks
Ian Barwick
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-11-04 13:27 ` Imseih (AWS), Sami <[email protected]>
2022-11-09 03:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-11-04 13:27 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Resubmitting patches with proper format.
Thanks
Sami Imseih
Amazon Web Services (AWS)
Attachments:
[application/octet-stream] v14-0002-Function-to-return-currently-vacuumed-or-cleaned.patch (10.3K, ../../[email protected]/2-v14-0002-Function-to-return-currently-vacuumed-or-cleaned.patch)
download | inline diff:
From 78a437946d5ccc70db98da9cc0fa0ae413afa625 Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Fri, 4 Nov 2022 07:52:02 -0500
Subject: [PATCH v14 2/2] Function to return currently vacuumed ( or cleaned )
index.
pg_stat_get_vacuum_index takes a PID as an argument and
returns the index relid of the currently vacuumed index.
NULL is returned if the backend is not involved in an index vacuum.
---
doc/src/sgml/func.sgml | 13 +++++++
doc/src/sgml/monitoring.sgml | 8 +++--
src/backend/access/heap/vacuumlazy.c | 9 +++++
src/backend/commands/vacuumparallel.c | 23 ++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 52 +++++++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 4 +++
src/include/commands/progress.h | 1 +
src/include/utils/backend_progress.h | 7 +++-
8 files changed, 114 insertions(+), 3 deletions(-)
22.2% doc/src/sgml/
11.2% src/backend/access/heap/
16.7% src/backend/commands/
37.1% src/backend/utils/adt/
5.7% src/include/catalog/
5.4% src/include/utils/
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 6e0425cb3d..39841354de 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -22670,6 +22670,19 @@ SELECT * FROM pg_ls_dir('.') WITH ORDINALITY AS t(ls,n);
parsing the text version.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_stat_get_vacuum_index</primary>
+ </indexterm>
+ <function>pg_stat_get_vacuum_index</function> ( <type>integer</type> )
+ <returnvalue>oid</returnvalue>
+ </para>
+ <para>
+ Returns the relid of an index current being vacuumed or cleaned by the backend.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index c61443f3d5..8d947d8c3a 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6517,7 +6517,9 @@ FROM pg_stat_get_backend_idset() AS backendid;
has been completely scanned. It may happen multiple times per vacuum
if <xref linkend="guc-maintenance-work-mem"/> (or, in the case of autovacuum,
<xref linkend="guc-autovacuum-work-mem"/> if set) is insufficient to store
- the number of dead tuples found.
+ the number of dead tuples found. While in this phase, the function
+ <function>pg_stat_get_vacuum_index()</function> will return the relid
+ of the index being vacuumed.
</entry>
</row>
<row>
@@ -6536,7 +6538,9 @@ FROM pg_stat_get_backend_idset() AS backendid;
<entry>
<command>VACUUM</command> is currently cleaning up indexes. This occurs after
the heap has been completely scanned and all vacuuming of the indexes
- and the heap has been completed.
+ and the heap has been completed. While in this phase, the function
+ <function>pg_stat_get_vacuum_index()</function> will return the relid
+ of the index being cleaned.
</entry>
</row>
<row>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 120a2b18d7..8cd8e34f88 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -350,6 +350,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
}
+ /* start the vacuum progress command and report to progress.h */
pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
RelationGetRelid(rel));
@@ -2341,6 +2342,9 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* report the index relid being vacuumed */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
@@ -2350,6 +2354,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
*/
pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
if (lazy_check_wraparound_failsafe(vacrel))
{
@@ -2705,6 +2710,9 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
Relation indrel = vacrel->indrels[idx];
IndexBulkDeleteResult *istat = vacrel->indstats[idx];
+ /* report the index relid being cleaned */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, RelationGetRelid(indrel));
+
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
@@ -2714,6 +2722,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
*/
pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
idx + 1);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
}
}
else
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 85e2de5d54..f2f51e86eb 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -840,6 +840,20 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
IndexBulkDeleteResult *istat = NULL;
IndexBulkDeleteResult *istat_res;
IndexVacuumInfo ivinfo;
+ Oid indrelid = RelationGetRelid(indrel);
+
+ /*
+ * If we are a parallel worker, start a PROGRESS_COMMAND_VACUUM_PARALLEL
+ * command for progress reporting and report the index being vacuumed.
+ */
+ if (IsParallelWorker())
+ {
+ pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM_PARALLEL,
+ pvs->shared->relid);
+ }
+
+ /* report the index being vacuumed or cleaned up */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, indrelid);
/*
* Update the pointer to the corresponding bulk-deletion result if someone
@@ -911,10 +925,19 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pfree(pvs->indname);
pvs->indname = NULL;
+ /*
+ * Reset the index relid begin vacuumed/cleaned
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDRELID, 0);
+
/*
* Update the number of indexes completed.
*/
pg_atomic_add_fetch_u32(&(pvs->shared->idx_completed_progress), 1);
+
+ /* if we are a parallel worker, end the command */
+ if (IsParallelWorker())
+ pgstat_progress_end_command();
}
/*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 96bffc0f2a..82eedae1f0 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -19,6 +19,7 @@
#include "access/xlogprefetcher.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
+#include "commands/progress.h"
#include "common/ip.h"
#include "funcapi.h"
#include "miscadmin.h"
@@ -2383,3 +2384,54 @@ pg_stat_have_stats(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(pgstat_have_entry(kind, dboid, objoid));
}
+
+/*
+ * For a given PID, return the relid of the index
+ * currently being vacuumed.
+ */
+Datum
+pg_stat_get_vacuum_index(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ int num_backends = pgstat_fetch_stat_numbackends();
+ int curr_backend;
+ Oid indrelid = 0;
+ ProgressCommandType cmdtype_vacuum = PROGRESS_COMMAND_VACUUM;
+ ProgressCommandType cmdtype_vacuum_parallel = PROGRESS_COMMAND_VACUUM_PARALLEL;
+
+ /* 1-based index */
+ for (curr_backend = 1; curr_backend <= num_backends; curr_backend++)
+ {
+ LocalPgBackendStatus *local_beentry;
+ PgBackendStatus *beentry;
+
+ local_beentry = pgstat_fetch_stat_local_beentry(curr_backend);
+ beentry = &local_beentry->backendStatus;
+
+ /*
+ * If a backend has a VACUUM or VACUUM_PARALLEL progress command type
+ * and the procpid matches the given PID, return the index relid
+ * that is current being vacuumed. In some cases, the index relid could
+ * be a '0' if we got here at the end of a vacuum or when no indexes
+ * are being vacuumed. In this case return NULL.
+ */
+ if (
+ (beentry->st_progress_command == cmdtype_vacuum
+ || beentry->st_progress_command == cmdtype_vacuum_parallel) &&
+ beentry->st_procpid == pid
+ )
+ {
+ indrelid = Int64GetDatum(beentry->st_progress_param[PROGRESS_VACUUM_INDRELID]);
+
+ if (indrelid > 0)
+ PG_RETURN_OID(indrelid);
+ else
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * No matching PID performing a VACUUM operation, return NULL.
+ */
+ PG_RETURN_NULL();
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 20f5aa56ea..dd5a32172a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11827,5 +11827,9 @@
proname => 'brin_minmax_multi_summary_send', provolatile => 's',
prorettype => 'bytea', proargtypes => 'pg_brin_minmax_multi_summary',
prosrc => 'brin_minmax_multi_summary_send' },
+{ oid => '4642',
+ descr => 'get the relid of the index being vacuumed by a backend',
+ proname => 'pg_stat_get_vacuum_index', provolatile => 'v', prorettype => 'int8',
+ proargtypes => 'int4', prosrc => 'pg_stat_get_vacuum_index' },
]
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 0e97c6d4ef..bc2b7d1e95 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -27,6 +27,7 @@
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
#define PROGRESS_VACUUM_INDEX_TOTAL 7
#define PROGRESS_VACUUM_INDEX_COMPLETED 8
+#define PROGRESS_VACUUM_INDRELID 9
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 47bf8029b0..74d9dfb4c7 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -17,6 +17,10 @@
/* ----------
* Command type for progress reporting purposes
+ *
+ * Note: PROGRESS_COMMAND_VACUUM_PARALLEL is not
+ * a command per se, but this type is used to track
+ * parallel vacuum workers progress.
* ----------
*/
typedef enum ProgressCommandType
@@ -27,7 +31,8 @@ typedef enum ProgressCommandType
PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
PROGRESS_COMMAND_BASEBACKUP,
- PROGRESS_COMMAND_COPY
+ PROGRESS_COMMAND_COPY,
+ PROGRESS_COMMAND_VACUUM_PARALLEL
} ProgressCommandType;
#define PGSTAT_NUM_PROGRESS_PARAM 20
--
2.32.1 (Apple Git-133)
[application/octet-stream] v14-0001-Add-2-new-columns-to-pg_stat_progress_vacuum.-Th.patch (24.5K, ../../[email protected]/3-v14-0001-Add-2-new-columns-to-pg_stat_progress_vacuum.-Th.patch)
download | inline diff:
From d0dba3c28dbf8915db4df731edb445e5064f97bb Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Fri, 4 Nov 2022 07:48:42 -0500
Subject: [PATCH v14 1/2] Add 2 new columns to pg_stat_progress_vacuum. The
columns are indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
contrib/bloom/blvacuum.c | 6 +++
doc/src/sgml/monitoring.sgml | 21 +++++++++
src/backend/access/brin/brin.c | 9 ++--
src/backend/access/gin/ginvacuum.c | 9 ++++
src/backend/access/gist/gistvacuum.c | 5 +++
src/backend/access/hash/hash.c | 8 +++-
src/backend/access/hash/hashpage.c | 4 +-
src/backend/access/heap/vacuumlazy.c | 40 ++++++++++++++++-
src/backend/access/nbtree/nbtree.c | 3 ++
src/backend/access/spgist/spgvacuum.c | 3 ++
src/backend/access/transam/parallel.c | 5 +++
src/backend/catalog/index.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 63 ++++++++++++++++++++++++++-
src/include/access/genam.h | 3 ++
src/include/access/hash.h | 3 +-
src/include/access/parallel.h | 2 +
src/include/commands/progress.h | 2 +
src/include/commands/vacuum.h | 2 +
src/test/regress/expected/rules.out | 4 +-
20 files changed, 184 insertions(+), 12 deletions(-)
10.5% doc/src/sgml/
5.8% src/backend/access/brin/
4.4% src/backend/access/gin/
6.2% src/backend/access/hash/
17.7% src/backend/access/heap/
7.5% src/backend/access/
32.8% src/backend/commands/
6.1% src/include/access/
5.6% src/
diff --git a/contrib/bloom/blvacuum.c b/contrib/bloom/blvacuum.c
index 91fae5b0c0..19ae819cfe 100644
--- a/contrib/bloom/blvacuum.c
+++ b/contrib/bloom/blvacuum.c
@@ -136,6 +136,9 @@ blbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
GenericXLogAbort(gxlogState);
}
UnlockReleaseBuffer(buffer);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/*
@@ -209,6 +212,9 @@ blvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
}
UnlockReleaseBuffer(buffer);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
IndexFreeSpaceMapVacuum(info->index);
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index e5d622d514..c61443f3d5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6455,6 +6455,27 @@ FROM pg_stat_get_backend_idset() AS backendid;
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes that wil be vacuumed. This value will be
+ <literal>0</literal> if there are no indexes to vacuum or
+ vacuum failsafe is triggered. See <xref linkend="guc-vacuum-failsafe-age"/>
+ for more on vacuum failsafe.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_completed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes vacuumed in the current vacuum cycle.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 20b7d65b94..bd96ecf41e 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -78,7 +78,7 @@ static void brinsummarize(Relation index, Relation heapRel, BlockNumber pageRang
static void form_and_insert_tuple(BrinBuildState *state);
static void union_tuples(BrinDesc *bdesc, BrinMemTuple *a,
BrinTuple *b);
-static void brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy);
+static void brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy, IndexVacuumInfo *info);
static bool add_values_to_range(Relation idxRel, BrinDesc *bdesc,
BrinMemTuple *dtup, Datum *values, bool *nulls);
static bool check_null_keys(BrinValues *bval, ScanKey *nullkeys, int nnullkeys);
@@ -952,7 +952,7 @@ brinvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
heapRel = table_open(IndexGetRelation(RelationGetRelid(info->index), false),
AccessShareLock);
- brin_vacuum_scan(info->index, info->strategy);
+ brin_vacuum_scan(info->index, info->strategy, info);
brinsummarize(info->index, heapRel, BRIN_ALL_BLOCKRANGES, false,
&stats->num_index_tuples, &stats->num_index_tuples);
@@ -1659,7 +1659,7 @@ union_tuples(BrinDesc *bdesc, BrinMemTuple *a, BrinTuple *b)
* and such.
*/
static void
-brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy)
+brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy, IndexVacuumInfo *info)
{
BlockNumber nblocks;
BlockNumber blkno;
@@ -1681,6 +1681,9 @@ brin_vacuum_scan(Relation idxrel, BufferAccessStrategy strategy)
brin_page_cleanup(idxrel, buf);
ReleaseBuffer(buf);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/*
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..3d5e4600dc 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -633,6 +633,9 @@ ginbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
UnlockReleaseBuffer(buffer);
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/* right now we found leftmost page in entry's BTree */
@@ -677,6 +680,9 @@ ginbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
LockBuffer(buffer, GIN_EXCLUSIVE);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
MemoryContextDelete(gvs.tmpCxt);
@@ -775,6 +781,9 @@ ginvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
}
UnlockReleaseBuffer(buffer);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/* Update the metapage with accurate page and entry counts */
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 0aa6e58a62..da3a0540d0 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -223,7 +223,12 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
break;
/* Iterate over pages, then loop back to recheck length */
for (; blkno < num_pages; blkno++)
+ {
gistvacuumpage(&vstate, blkno, blkno);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
+ }
}
/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index c361509d68..14790adf61 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -550,7 +550,7 @@ loop_top:
cachedmetap->hashm_highmask,
cachedmetap->hashm_lowmask, &tuples_removed,
&num_index_tuples, split_cleanup,
- callback, callback_state);
+ callback, callback_state, info);
_hash_dropbuf(rel, bucket_buf);
@@ -686,7 +686,8 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
uint32 maxbucket, uint32 highmask, uint32 lowmask,
double *tuples_removed, double *num_index_tuples,
bool split_cleanup,
- IndexBulkDeleteCallback callback, void *callback_state)
+ IndexBulkDeleteCallback callback, void *callback_state,
+ IndexVacuumInfo *info)
{
BlockNumber blkno;
Buffer buf;
@@ -775,6 +776,9 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
if (num_index_tuples)
*num_index_tuples += 1;
}
+
+ if (info && info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
/* retain the pin on primary bucket page till end of bucket scan */
diff --git a/src/backend/access/hash/hashpage.c b/src/backend/access/hash/hashpage.c
index d2edcd4617..e9ce1a1110 100644
--- a/src/backend/access/hash/hashpage.c
+++ b/src/backend/access/hash/hashpage.c
@@ -759,7 +759,7 @@ restart_expand:
hashbucketcleanup(rel, old_bucket, buf_oblkno, start_oblkno, NULL,
maxbucket, highmask, lowmask, NULL, NULL, true,
- NULL, NULL);
+ NULL, NULL, NULL);
_hash_dropbuf(rel, buf_oblkno);
@@ -1327,7 +1327,7 @@ _hash_splitbucket(Relation rel,
hashbucketcleanup(rel, obucket, bucket_obuf,
BufferGetBlockNumber(bucket_obuf), NULL,
maxbucket, highmask, lowmask, NULL, NULL, true,
- NULL, NULL);
+ NULL, NULL, NULL);
}
else
{
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index dfbe37472f..120a2b18d7 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -420,6 +420,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+
+ /* report number of indexes to vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2341,6 +2345,12 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /*
+ * Done vacuuming an index. Increment the indexes completed
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2384,6 +2394,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_VACUUMS,
vacrel->num_index_scans);
+ /*
+ * Reset the indexes completed at this point.
+ * If we end up in another index vacuum cycle, we will
+ * start counting from the start.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
+
return allindexes;
}
@@ -2633,10 +2650,17 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
{
vacrel->failsafe_active = true;
- /* Disable index vacuuming, index cleanup, and heap rel truncation */
+ /*
+ * Disable index vacuuming, index cleanup, and heap rel truncation
+ *
+ * Also, report to progress.h that we are no longer tracking
+ * index vacuum/cleanup.
+ */
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2684,6 +2708,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /*
+ * Done cleaning an index. Increment the indexes completed
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
}
}
else
@@ -2693,6 +2723,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->num_index_scans,
estimated_count);
}
+
+ /*
+ * Reset the indexes completed at this point.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
+
}
/*
@@ -2718,6 +2754,7 @@ lazy_vacuum_one_index(Relation indrel, IndexBulkDeleteResult *istat,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = true;
ivinfo.message_level = DEBUG2;
ivinfo.num_heap_tuples = reltuples;
@@ -2766,6 +2803,7 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = estimated_count;
ivinfo.message_level = DEBUG2;
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index b52eca8f38..9904a9eb0c 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -998,6 +998,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
if (info->report_progress)
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
scanblkno);
+
+ if (info->report_parallel_progress)
+ info->parallel_progress_callback(info->parallel_progress_arg);
}
}
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..b1ab491980 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -843,6 +843,9 @@ spgvacuumscan(spgBulkDeleteState *bds)
/* empty the pending-list after each page */
if (bds->pendingList != NULL)
spgprocesspending(bds);
+
+ if (bds->info->report_parallel_progress)
+ bds->info->parallel_progress_callback(bds->info->parallel_progress_arg);
}
}
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index ee0985c7ed..0b4bde8297 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -185,6 +185,8 @@ CreateParallelContext(const char *library_name, const char *function_name,
pcxt->library_name = pstrdup(library_name);
pcxt->function_name = pstrdup(function_name);
pcxt->error_context_stack = error_context_stack;
+ pcxt->parallel_progress_callback = NULL;
+ pcxt->parallel_progress_arg = NULL;
shm_toc_initialize_estimator(&pcxt->estimator);
dlist_push_head(&pcxt_list, &pcxt->node);
@@ -785,6 +787,9 @@ WaitForParallelWorkersToFinish(ParallelContext *pcxt)
*/
CHECK_FOR_INTERRUPTS();
+ if (pcxt->parallel_progress_callback)
+ pcxt->parallel_progress_callback(pcxt->parallel_progress_arg);
+
for (i = 0; i < pcxt->nworkers_launched; ++i)
{
/*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 61f1d3926a..11b32129a7 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -3348,6 +3348,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
ivinfo.index = indexRelation;
ivinfo.analyze_only = false;
ivinfo.report_progress = true;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = true;
ivinfo.message_level = DEBUG2;
ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2d8104b090..c37b20b91b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1165,7 +1165,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_completed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index f26d796e52..85e2de5d54 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -30,6 +30,7 @@
#include "access/table.h"
#include "access/xact.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -103,6 +104,17 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /*
+ * Counter for vacuuming and cleanup progress reporting.
+ * This value is used to report index vacuum/cleanup progress
+ * in parallel_vacuum_progress_report. We keep this
+ * counter to avoid having to loop through
+ * ParallelVacuumState->indstats to determine the number
+ * of indexes completed.
+ */
+ pg_atomic_uint32 idx_completed_progress;
+
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -273,6 +285,9 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
Assert(pcxt->nworkers > 0);
pvs->pcxt = pcxt;
+ pvs->pcxt->parallel_progress_callback = parallel_vacuum_progress_report;
+ pvs->pcxt->parallel_progress_arg = pvs;
+
/* Estimate size for index vacuum stats -- PARALLEL_VACUUM_KEY_INDEX_STATS */
est_indstats_len = mul_size(sizeof(PVIndStats), nindexes);
shm_toc_estimate_chunk(&pcxt->estimator, est_indstats_len);
@@ -364,6 +379,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
pg_atomic_init_u32(&(shared->idx), 0);
+ pg_atomic_init_u32(&(shared->idx_completed_progress), 0);
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
@@ -618,8 +634,9 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
vacuum));
}
- /* Reset the parallel index processing counter */
+ /* Reset the parallel index processing counter ( index progress counter also ) */
pg_atomic_write_u32(&(pvs->shared->idx), 0);
+ pg_atomic_write_u32(&(pvs->shared->idx_completed_progress), 0);
/* Setup the shared cost-based vacuum delay and launch workers */
if (nworkers > 0)
@@ -834,10 +851,13 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = true;
ivinfo.message_level = DEBUG2;
ivinfo.estimated_count = pvs->shared->estimated_count;
ivinfo.num_heap_tuples = pvs->shared->reltuples;
ivinfo.strategy = pvs->bstrategy;
+ ivinfo.parallel_progress_callback = parallel_vacuum_progress_report;
+ ivinfo.parallel_progress_arg = pvs;
/* Update error traceback information */
pvs->indname = pstrdup(RelationGetRelationName(indrel));
@@ -846,9 +866,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
switch (indstats->status)
{
case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_VACUUM_INDEX);
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
break;
case PARALLEL_INDVAC_STATUS_NEED_CLEANUP:
+ pgstat_progress_update_param(PROGRESS_VACUUM_PHASE, PROGRESS_VACUUM_PHASE_INDEX_CLEANUP);
istat_res = vac_cleanup_one_index(&ivinfo, istat);
break;
default:
@@ -888,6 +910,11 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
pvs->indname = NULL;
+
+ /*
+ * Update the number of indexes completed.
+ */
+ pg_atomic_add_fetch_u32(&(pvs->shared->idx_completed_progress), 1);
}
/*
@@ -1072,3 +1099,37 @@ parallel_vacuum_error_callback(void *arg)
return;
}
}
+
+/*
+ * Read the number of indexes vacuumed from the shared counter
+ * and report it to progress.h
+ *
+ * This function is only called by the leader process of
+ * a parallel vacuum.
+ *
+ * If report_parallel_progress is set to true,
+ * this function is called from major loops inside the
+ * ambulkdelete and amvacuumcleanup.
+ *
+ * This function is also called inside WaitForParallelWorkersToFinish
+ * when report_parallel_progress is set to true.
+ *
+ * The reason for calling this function in both vacuum AM's
+ * and WaitForParallelWorkersToFinish is to ensure that
+ * parallel vacuum progress is constantly being reported
+ * if the leader process is either waiting for parallel
+ * proceses to finish (WaitForParallelWorkersToFinish) or
+ * the leader process is the last to finish and still
+ * inside the vacuum AM's.
+ */
+void
+parallel_vacuum_progress_report(void *arg)
+{
+ ParallelVacuumState *pvs = (ParallelVacuumState *) arg;
+
+ if (IsParallelWorker())
+ return;
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(pvs->shared->idx_completed_progress)));
+}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index e1c4fdbd03..6d5fc189f4 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -46,10 +46,13 @@ typedef struct IndexVacuumInfo
Relation index; /* the index being vacuumed */
bool analyze_only; /* ANALYZE (without any actual vacuum) */
bool report_progress; /* emit progress.h status reports */
+ bool report_parallel_progress; /* emit progress.h status reports for parallel vacuum */
bool estimated_count; /* num_heap_tuples is an estimate */
int message_level; /* ereport level for progress messages */
double num_heap_tuples; /* tuples remaining in heap */
BufferAccessStrategy strategy; /* access strategy for reads */
+ void (*parallel_progress_callback)(void *arg);
+ void *parallel_progress_arg;
} IndexVacuumInfo;
/*
diff --git a/src/include/access/hash.h b/src/include/access/hash.h
index da372841c4..14fb75a4ad 100644
--- a/src/include/access/hash.h
+++ b/src/include/access/hash.h
@@ -480,6 +480,7 @@ extern void hashbucketcleanup(Relation rel, Bucket cur_bucket,
uint32 maxbucket, uint32 highmask, uint32 lowmask,
double *tuples_removed, double *num_index_tuples,
bool split_cleanup,
- IndexBulkDeleteCallback callback, void *callback_state);
+ IndexBulkDeleteCallback callback, void *callback_state,
+ IndexVacuumInfo *info);
#endif /* HASH_H */
diff --git a/src/include/access/parallel.h b/src/include/access/parallel.h
index 1ec8e33af4..77e9c0a517 100644
--- a/src/include/access/parallel.h
+++ b/src/include/access/parallel.h
@@ -46,6 +46,8 @@ typedef struct ParallelContext
ParallelWorkerInfo *worker;
int nknown_attached_workers;
bool *known_attached_workers;
+ void (*parallel_progress_callback)(void *arg);
+ void *parallel_progress_arg;
} ParallelContext;
typedef struct ParallelWorkerContext
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..0e97c6d4ef 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_INDEX_TOTAL 7
+#define PROGRESS_VACUUM_INDEX_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5d816ba7f4..9510d028d3 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -338,4 +338,6 @@ extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+extern void parallel_vacuum_progress_report(void *arg);
+
#endif /* VACUUM_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 624d0e5aae..e0bf81247f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2018,7 +2018,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_completed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT s.stats_reset,
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-04 13:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-11-09 03:00 ` Andres Freund <[email protected]>
2022-11-11 19:10 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Andres Freund @ 2022-11-09 03:00 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Hi,
On 2022-11-04 13:27:34 +0000, Imseih (AWS), Sami wrote:
> diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
> index b4fa5f6bf8..3d5e4600dc 100644
> --- a/src/backend/access/gin/ginvacuum.c
> +++ b/src/backend/access/gin/ginvacuum.c
> @@ -633,6 +633,9 @@ ginbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
> UnlockReleaseBuffer(buffer);
> buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
> RBM_NORMAL, info->strategy);
> +
> + if (info->report_parallel_progress)
> + info->parallel_progress_callback(info->parallel_progress_arg);
> }
>
> /* right now we found leftmost page in entry's BTree */
I don't think any of these progress callbacks should be done while pinning a
buffer and ...
> @@ -677,6 +680,9 @@ ginbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
> buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
> RBM_NORMAL, info->strategy);
> LockBuffer(buffer, GIN_EXCLUSIVE);
> +
> + if (info->report_parallel_progress)
> + info->parallel_progress_callback(info->parallel_progress_arg);
> }
>
> MemoryContextDelete(gvs.tmpCxt);
... definitely not while holding a buffer lock.
I also don't understand why info->parallel_progress_callback exists? It's only
set to parallel_vacuum_progress_report(). Why make this stuff more expensive
than it has to already be?
> +parallel_vacuum_progress_report(void *arg)
> +{
> + ParallelVacuumState *pvs = (ParallelVacuumState *) arg;
> +
> + if (IsParallelWorker())
> + return;
> +
> + pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
> + pg_atomic_read_u32(&(pvs->shared->idx_completed_progress)));
> +}
So each of the places that call this need to make an additional external
function call for each page, just to find that there's nothing to do or to
make yet another indirect function call. This should probably a static inline
function.
This is called, for every single page, just to read the number of indexes
completed by workers? A number that barely ever changes?
This seems all wrong.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-04 13:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-09 03:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
@ 2022-11-11 19:10 ` Imseih (AWS), Sami <[email protected]>
2022-11-18 13:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-11-11 19:10 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> I don't think any of these progress callbacks should be done while pinning a
> buffer and ...
Good point.
> I also don't understand why info->parallel_progress_callback exists? It's only
> set to parallel_vacuum_progress_report(). Why make this stuff more expensive
> than it has to already be?
Agree. Modified the patch to avoid the callback .
> So each of the places that call this need to make an additional external
> function call for each page, just to find that there's nothing to do or to
> make yet another indirect function call. This should probably a static inline
> function.
Even better to just remove a function call altogether.
> This is called, for every single page, just to read the number of indexes
> completed by workers? A number that barely ever changes?
I will take the initial suggestion by Sawada-san to update the progress
every 1GB of blocks scanned.
Also, It sems to me that we don't need to track progress in brin indexes,
Since it is rare, if ever, this type of index will go through very heavy
block scans. In fact, I noticed the vacuum AMs for brin don't invoke the
vacuum_delay_point at all.
The attached patch addresses the feedback.
Regards,
Sami Imseih
Amazon Web Services (AWS)
Attachments:
[application/octet-stream] v15-0001-Add-2-new-columns-to-pg_stat_progress_vacuum.-Th.patch (21.4K, ../../[email protected]/2-v15-0001-Add-2-new-columns-to-pg_stat_progress_vacuum.-Th.patch)
download | inline diff:
From c59171a3ecee410380e9f62bfc2d678243842a2c Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)"
<[email protected]>
Date: Fri, 11 Nov 2022 18:40:14 +0000
Subject: [PATCH 1/1] Add 2 new columns to pg_stat_progress_vacuum. The columns
are indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
contrib/bloom/blvacuum.c | 9 ++++
doc/src/sgml/monitoring.sgml | 21 +++++++++
src/backend/access/gin/ginvacuum.c | 10 ++++
src/backend/access/gist/gistvacuum.c | 7 +++
src/backend/access/hash/hash.c | 7 +++
src/backend/access/heap/vacuumlazy.c | 40 +++++++++++++++-
src/backend/access/nbtree/nbtree.c | 3 ++
src/backend/access/spgist/spgvacuum.c | 6 +++
src/backend/catalog/index.c | 2 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuumparallel.c | 66 +++++++++++++++++++++++++--
src/include/access/genam.h | 2 +
src/include/commands/progress.h | 2 +
src/include/commands/vacuum.h | 6 +++
src/test/regress/expected/rules.out | 4 +-
15 files changed, 182 insertions(+), 6 deletions(-)
diff --git a/contrib/bloom/blvacuum.c b/contrib/bloom/blvacuum.c
index 91fae5b0c0..92a557518f 100644
--- a/contrib/bloom/blvacuum.c
+++ b/contrib/bloom/blvacuum.c
@@ -15,12 +15,14 @@
#include "access/genam.h"
#include "bloom.h"
#include "catalog/storage.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
+#include "utils/backend_progress.h"
/*
@@ -62,6 +64,10 @@ blbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
*itupEnd;
vacuum_delay_point();
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(info->idx_completed_progress)));
+
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
@@ -192,6 +198,9 @@ blvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
Page page;
vacuum_delay_point();
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(info->idx_completed_progress)));
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index e5d622d514..c61443f3d5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6455,6 +6455,27 @@ FROM pg_stat_get_backend_idset() AS backendid;
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes that wil be vacuumed. This value will be
+ <literal>0</literal> if there are no indexes to vacuum or
+ vacuum failsafe is triggered. See <xref linkend="guc-vacuum-failsafe-age"/>
+ for more on vacuum failsafe.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_completed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes vacuumed in the current vacuum cycle.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6bf8..623ee78afa 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,12 +17,14 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "storage/predicate.h"
+#include "utils/backend_progress.h"
#include "utils/memutils.h"
struct GinVacuumState
@@ -665,6 +667,10 @@ ginbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
vacuum_delay_point();
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(info->idx_completed_progress)));
+
for (i = 0; i < nRoot; i++)
{
ginVacuumPostingTree(&gvs, rootOfPostingTree[i]);
@@ -751,6 +757,10 @@ ginvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
vacuum_delay_point();
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(info->idx_completed_progress)));
+
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
LockBuffer(buffer, GIN_SHARE);
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 0aa6e58a62..45902e7aa9 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,11 +17,13 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
+#include "utils/backend_progress.h"
#include "utils/memutils.h"
/* Working state needed by gistbulkdelete */
@@ -223,7 +225,12 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
break;
/* Iterate over pages, then loop back to recheck length */
for (; blkno < num_pages; blkno++)
+ {
gistvacuumpage(&vstate, blkno, blkno);
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(info->idx_completed_progress)));
+ }
}
/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index c361509d68..798c93b6cb 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -505,6 +505,13 @@ loop_top:
blkno = bucket_blkno;
+ /*
+ * For hash indexes, we report parallel vacuum progress
+ * for every bucket.
+ */
+ if (info->report_parallel_progress)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(info->idx_completed_progress)));
/*
* We need to acquire a cleanup lock on the primary bucket page to out
* wait concurrent scans before deleting the dead tuples.
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index dfbe37472f..120a2b18d7 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -420,6 +420,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+
+ /* report number of indexes to vacuum */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, vacrel->nindexes);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -2341,6 +2345,12 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /*
+ * Done vacuuming an index. Increment the indexes completed
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2384,6 +2394,13 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_VACUUMS,
vacrel->num_index_scans);
+ /*
+ * Reset the indexes completed at this point.
+ * If we end up in another index vacuum cycle, we will
+ * start counting from the start.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
+
return allindexes;
}
@@ -2633,10 +2650,17 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
{
vacrel->failsafe_active = true;
- /* Disable index vacuuming, index cleanup, and heap rel truncation */
+ /*
+ * Disable index vacuuming, index cleanup, and heap rel truncation
+ *
+ * Also, report to progress.h that we are no longer tracking
+ * index vacuum/cleanup.
+ */
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2684,6 +2708,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /*
+ * Done cleaning an index. Increment the indexes completed
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
}
}
else
@@ -2693,6 +2723,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->num_index_scans,
estimated_count);
}
+
+ /*
+ * Reset the indexes completed at this point.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
+
}
/*
@@ -2718,6 +2754,7 @@ lazy_vacuum_one_index(Relation indrel, IndexBulkDeleteResult *istat,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = true;
ivinfo.message_level = DEBUG2;
ivinfo.num_heap_tuples = reltuples;
@@ -2766,6 +2803,7 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = estimated_count;
ivinfo.message_level = DEBUG2;
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index b52eca8f38..374a712952 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -998,6 +998,9 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
if (info->report_progress)
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
scanblkno);
+ if (info->report_parallel_progress && (scanblkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(info->idx_completed_progress)));
}
}
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 0049630532..1f13bc28ce 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,11 +21,13 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
+#include "utils/backend_progress.h"
#include "utils/snapmgr.h"
@@ -843,6 +845,10 @@ spgvacuumscan(spgBulkDeleteState *bds)
/* empty the pending-list after each page */
if (bds->pendingList != NULL)
spgprocesspending(bds);
+ /* report parallel vacuum progress */
+ if (bds->info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(bds->info->idx_completed_progress)));
}
}
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 61f1d3926a..3728750569 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -3348,10 +3348,12 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
ivinfo.index = indexRelation;
ivinfo.analyze_only = false;
ivinfo.report_progress = true;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = true;
ivinfo.message_level = DEBUG2;
ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
ivinfo.strategy = NULL;
+ pg_atomic_init_u32(&(ivinfo.idx_completed_progress), 0);
/*
* Encode TIDs as int8 values for the sort, rather than directly sorting
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2d8104b090..c37b20b91b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1165,7 +1165,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_completed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index f26d796e52..c3579af169 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -30,6 +30,7 @@
#include "access/table.h"
#include "access/xact.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -103,6 +104,17 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /*
+ * Counter for vacuuming and cleanup progress reporting.
+ * This value is used to report index vacuum/cleanup progress
+ * in parallel_vacuum_progress_report. We keep this
+ * counter to avoid having to loop through
+ * ParallelVacuumState->indstats to determine the number
+ * of indexes completed.
+ */
+ pg_atomic_uint32 idx_completed_progress;
+
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -213,6 +225,7 @@ static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation
static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_index_scans,
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
+static void parallel_wait_for_workers_to_finish(ParallelVacuumState *pvs);
/*
* Try to enter parallel mode and create a parallel context. Then initialize
@@ -364,6 +377,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
pg_atomic_init_u32(&(shared->idx), 0);
+ pg_atomic_init_u32(&(shared->idx_completed_progress), 0);
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
@@ -618,8 +632,9 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
vacuum));
}
- /* Reset the parallel index processing counter */
+ /* Reset the parallel index processing counter ( index progress counter also ) */
pg_atomic_write_u32(&(pvs->shared->idx), 0);
+ pg_atomic_write_u32(&(pvs->shared->idx_completed_progress), 0);
/* Setup the shared cost-based vacuum delay and launch workers */
if (nworkers > 0)
@@ -688,7 +703,21 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
*/
if (nworkers > 0)
{
- /* Wait for all vacuum workers to finish */
+ /*
+ * To wait for parallel workers to finish,
+ * first call parallel_wait_for_workers_to_finish
+ * which is responsible for reporting the
+ * number of indexes completed.
+ *
+ * Afterwards, WaitForParallelWorkersToFinish is called
+ * to do the real work of waiting for parallel workers
+ * to finish.
+ *
+ * Note: Both routines will acquire a WaitLatch in their
+ * respective loops.
+ */
+ parallel_wait_for_workers_to_finish(pvs);
+
WaitForParallelWorkersToFinish(pvs->pcxt);
for (int i = 0; i < pvs->pcxt->nworkers_launched; i++)
@@ -838,7 +867,12 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
ivinfo.estimated_count = pvs->shared->estimated_count;
ivinfo.num_heap_tuples = pvs->shared->reltuples;
ivinfo.strategy = pvs->bstrategy;
-
+ ivinfo.idx_completed_progress = pvs->shared->idx_completed_progress;
+ /* Only the leader should report parallel vacuum progress */
+ if (!IsParallelWorker())
+ ivinfo.report_parallel_progress = true;
+ else
+ ivinfo.report_parallel_progress = false;
/* Update error traceback information */
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
@@ -857,6 +891,9 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
RelationGetRelationName(indrel));
}
+ if (ivinfo.report_parallel_progress)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(ivinfo.idx_completed_progress)));
/*
* Copy the index bulk-deletion result returned from ambulkdelete and
* amvacuumcleanup to the DSM segment if it's the first cycle because they
@@ -888,6 +925,9 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
pvs->indname = NULL;
+
+ /* Update the number of indexes completed. */
+ pg_atomic_add_fetch_u32(&(pvs->shared->idx_completed_progress), 1);
}
/*
@@ -1072,3 +1112,23 @@ parallel_vacuum_error_callback(void *arg)
return;
}
}
+
+/*
+ * Check if we are done vacuuming indexes and report
+ * progress.
+ *
+ * We nap using with a WaitLatch to avoid a busy loop.
+ */
+void
+parallel_wait_for_workers_to_finish(ParallelVacuumState *pvs)
+{
+ while (pg_atomic_read_u32(&(pvs->shared->idx_completed_progress)) < pvs->nindexes)
+ {
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ pg_atomic_read_u32(&(pvs->shared->idx_completed_progress)));
+
+ (void) WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_PARALLEL_FINISH);
+ ResetLatch(MyLatch);
+ }
+}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index e1c4fdbd03..bf69c93d31 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -46,10 +46,12 @@ typedef struct IndexVacuumInfo
Relation index; /* the index being vacuumed */
bool analyze_only; /* ANALYZE (without any actual vacuum) */
bool report_progress; /* emit progress.h status reports */
+ bool report_parallel_progress; /* emit progress.h status reports for parallel vacuum */
bool estimated_count; /* num_heap_tuples is an estimate */
int message_level; /* ereport level for progress messages */
double num_heap_tuples; /* tuples remaining in heap */
BufferAccessStrategy strategy; /* access strategy for reads */
+ pg_atomic_uint32 idx_completed_progress;
} IndexVacuumInfo;
/*
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938caf4..0e97c6d4ef 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_INDEX_TOTAL 7
+#define PROGRESS_VACUUM_INDEX_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5d816ba7f4..34cf36c425 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -64,6 +64,12 @@
/* value for checking vacuum flags */
#define VACUUM_OPTION_MAX_VALID_VALUE ((1 << 3) - 1)
+/*
+ * Parallel Index vacuum progress is reported every 1GB of blocks
+ * scanned.
+ */
+#define REPORT_PARALLEL_VACUUM_EVERY_PAGES ((BlockNumber) (((uint64) 1024 * 1024 * 1024) / BLCKSZ))
+
/* Abstract type for parallel vacuum state */
typedef struct ParallelVacuumState ParallelVacuumState;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 624d0e5aae..e0bf81247f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2018,7 +2018,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_completed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT s.stats_reset,
--
2.37.1
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-04 13:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-09 03:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-11-11 19:10 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-11-18 13:07 ` Masahiko Sawada <[email protected]>
2022-11-28 23:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
0 siblings, 1 reply; 74+ messages in thread
From: Masahiko Sawada @ 2022-11-18 13:07 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Andres Freund <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Sat, Nov 12, 2022 at 4:10 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> > I don't think any of these progress callbacks should be done while pinning a
> > buffer and ...
>
> Good point.
>
> > I also don't understand why info->parallel_progress_callback exists? It's only
> > set to parallel_vacuum_progress_report(). Why make this stuff more expensive
> > than it has to already be?
>
> Agree. Modified the patch to avoid the callback .
>
> > So each of the places that call this need to make an additional external
> > function call for each page, just to find that there's nothing to do or to
> > make yet another indirect function call. This should probably a static inline
> > function.
>
> Even better to just remove a function call altogether.
>
> > This is called, for every single page, just to read the number of indexes
> > completed by workers? A number that barely ever changes?
>
> I will take the initial suggestion by Sawada-san to update the progress
> every 1GB of blocks scanned.
>
> Also, It sems to me that we don't need to track progress in brin indexes,
> Since it is rare, if ever, this type of index will go through very heavy
> block scans. In fact, I noticed the vacuum AMs for brin don't invoke the
> vacuum_delay_point at all.
>
> The attached patch addresses the feedback.
>
Thank you for updating the patch! Here are review comments on v15 patch:
+ <para>
+ Number of indexes that wil be vacuumed. This value will be
+ <literal>0</literal> if there are no indexes to vacuum or
+ vacuum failsafe is triggered.
I think that indexes_total should be 0 also when INDEX_CLEANUP is off.
---
+ /*
+ * Reset the indexes completed at this point.
+ * If we end up in another index vacuum cycle, we will
+ * start counting from the start.
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
I think we don't need to reset it at the end of index vacuuming. There
is a small window before switching to the next phase. If we reset this
value while showing "index vacuuming" phase, the user might get
confused. Instead, we can reset it at the beginning of the index
vacuuming.
---
+void
+parallel_wait_for_workers_to_finish(ParallelVacuumState *pvs)
+{
+ while (pg_atomic_read_u32(&(pvs->shared->idx_completed_progress))
< pvs->nindexes)
+ {
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+
pg_atomic_read_u32(&(pvs->shared->idx_completed_progress)));
+
+ (void) WaitLatch(MyLatch, WL_LATCH_SET |
WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_PARALLEL_FINISH);
+ ResetLatch(MyLatch);
+ }
+}
We should add CHECK_FOR_INTERRUPTS() at the beginning of the loop to
make the wait interruptible.
I think it would be better to update the counter only when the value
has been increased.
I think we should set a timeout, say 1 sec, to WaitLatch so that it
can periodically check the progress.
Probably it's better to have a new wait event for this wait in order
to distinguish wait for workers to complete index vacuum from the wait
for workers to exit.
---
@@ -838,7 +867,12 @@
parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation
indrel,
ivinfo.estimated_count = pvs->shared->estimated_count;
ivinfo.num_heap_tuples = pvs->shared->reltuples;
ivinfo.strategy = pvs->bstrategy;
-
+ ivinfo.idx_completed_progress = pvs->shared->idx_completed_progress;
and
@@ -998,6 +998,9 @@ btvacuumscan(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats,
if (info->report_progress)
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
scanblkno);
+ if (info->report_parallel_progress &&
(scanblkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+
pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+
pg_atomic_read_u32(&(info->idx_completed_progress)));
}
I think this doesn't work, since ivinfo.idx_completed is in the
backend-local memory. Instead, I think we can have a function in
vacuumparallel.c that updates the progress. Then we can have index AM
call this function.
---
+ if (!IsParallelWorker())
+ ivinfo.report_parallel_progress = true;
+ else
+ ivinfo.report_parallel_progress = false;
We can do like:
ivinfo.report_parallel_progress = !IsParallelWorker();
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-04 13:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-09 03:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-11-11 19:10 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-18 13:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
@ 2022-11-28 23:57 ` Imseih (AWS), Sami <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-11-28 23:57 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Andres Freund <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> I think that indexes_total should be 0 also when INDEX_CLEANUP is off.
Patch updated for handling of INDEX_CLEANUP = off, with an update to
the documentation as well.
> I think we don't need to reset it at the end of index vacuuming. There
> is a small window before switching to the next phase. If we reset this
> value while showing "index vacuuming" phase, the user might get
> confused. Instead, we can reset it at the beginning of the index
> vacuuming.
No, I think the way it's currently done is correct. We want to reset the number
of indexes completed before we increase the num_index_scans ( index vacuum cycle ).
This ensures that when we enter a new index cycle, the number of indexes completed
are already reset. The 2 fields that matter here is how many indexes are vacuumed in the
currently cycle and this is called out in the documentation as such.
> We should add CHECK_FOR_INTERRUPTS() at the beginning of the loop to
> make the wait interruptible.
Done.
> I think it would be better to update the counter only when the value
> has been increased.
Done. Did so by checking the progress value from the beentry directly
in the function. We can do a more generalized
> I think we should set a timeout, say 1 sec, to WaitLatch so that it
> can periodically check the progress.
Done.
> Probably it's better to have a new wait event for this wait in order
> to distinguish wait for workers to complete index vacuum from the wait
> for workers to exit.
I was trying to avoid introducing a new wait event, but thinking about it,
I agree with your suggestion.
I created a new ParallelVacuumFinish wait event and documentation
for the wait event.
> I think this doesn't work, since ivinfo.idx_completed is in the
> backend-local memory. Instead, I think we can have a function in
> vacuumparallel.c that updates the progress. Then we can have index AM
> call this function.
Yeah, you're absolutely correct.
To make this work correctly, I did something similar to VacuumActiveNWorkers
and introduced an extern variable called ParallelVacuumProgress.
This variable points to pvs->shared->idx_completed_progress.
The index AMs then call parallel_vacuum_update_progress which
Is responsible for updating the progress with the current value
of ParallelVacuumProgress.
ParallelVacuumProgress is also set to NULL at the end of every index cycle.
> ivinfo.report_parallel_progress = !IsParallelWorker();
Done
Regards,
Sami Imseih
Amazon Web Services (AWS)
Attachments:
[application/octet-stream] v16-0001-Add-2-new-columns-to-pg_stat_progress_vacuum.-Th.patch (25.4K, ../../[email protected]/2-v16-0001-Add-2-new-columns-to-pg_stat_progress_vacuum.-Th.patch)
download | inline diff:
From db03efc9f21fa8cd9597cff146bbcce0becdf86a Mon Sep 17 00:00:00 2001
From: "Imseih (AWS)" <[email protected]>
Date: Mon, 28 Nov 2022 17:47:04 -0600
Subject: [PATCH 1/1] Add 2 new columns to pg_stat_progress_vacuum. The columns
are indexes_total as the total indexes to be vacuumed or cleaned and
indexes_processed as the number of indexes vacuumed or cleaned up so far.
Author: Sami Imseih, based on suggestions by Nathan Bossart, Peter Geoghegan and Masahiko Sawada
Reviewed by: Nathan Bossart, Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/[email protected]
---
contrib/bloom/blvacuum.c | 7 ++
doc/src/sgml/monitoring.sgml | 26 ++++++
src/backend/access/gin/ginvacuum.c | 8 ++
src/backend/access/gist/gistvacuum.c | 6 ++
src/backend/access/hash/hash.c | 6 ++
src/backend/access/heap/vacuumlazy.c | 33 ++++++-
src/backend/access/nbtree/nbtree.c | 2 +
src/backend/access/spgist/spgvacuum.c | 5 ++
src/backend/catalog/index.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/vacuum.c | 2 +
src/backend/commands/vacuumparallel.c | 110 +++++++++++++++++++++++-
src/backend/utils/activity/wait_event.c | 3 +
src/include/access/genam.h | 1 +
src/include/commands/progress.h | 2 +
src/include/commands/vacuum.h | 9 ++
src/include/utils/wait_event.h | 3 +-
src/test/regress/expected/rules.out | 4 +-
18 files changed, 223 insertions(+), 8 deletions(-)
diff --git a/contrib/bloom/blvacuum.c b/contrib/bloom/blvacuum.c
index 91fae5b..023fc7e 100644
--- a/contrib/bloom/blvacuum.c
+++ b/contrib/bloom/blvacuum.c
@@ -15,12 +15,14 @@
#include "access/genam.h"
#include "bloom.h"
#include "catalog/storage.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
#include "postmaster/autovacuum.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
+#include "utils/backend_progress.h"
/*
@@ -62,6 +64,9 @@ blbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
*itupEnd;
vacuum_delay_point();
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ parallel_vacuum_update_progress();
+
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
@@ -192,6 +197,8 @@ blvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
Page page;
vacuum_delay_point();
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ parallel_vacuum_update_progress();
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 5579b8b..477dfe2 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1759,6 +1759,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry><literal>ParallelFinish</literal></entry>
<entry>Waiting for parallel workers to finish computing.</entry>
</row>
+ <row>
+ <entry><literal>ParallelVacuumFinish</literal></entry>
+ <entry>Waiting for parallel vacuum workers to finish computing.</entry>
+ </row>
<row>
<entry><literal>ProcArrayGroupUpdate</literal></entry>
<entry>Waiting for the group leader to clear the transaction ID at
@@ -6815,6 +6819,28 @@ FROM pg_stat_get_backend_idset() AS backendid;
Number of dead tuples collected since the last index vacuum cycle.
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes that wil be vacuumed. This value will be
+ <literal>0</literal> if there are no indexes to vacuum, <literal>INDEX_CLEANUP</literal>
+ is set to <literal>OFF</literal>, or vacuum failsafe is triggered.
+ See <xref linkend="guc-vacuum-failsafe-age"/>
+ for more on vacuum failsafe.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>indexes_completed</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes vacuumed in the current vacuum cycle.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index b4fa5f6..e681164 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -17,12 +17,14 @@
#include "access/gin_private.h"
#include "access/ginxlog.h"
#include "access/xloginsert.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
#include "postmaster/autovacuum.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
#include "storage/predicate.h"
+#include "utils/backend_progress.h"
#include "utils/memutils.h"
struct GinVacuumState
@@ -665,6 +667,9 @@ ginbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
vacuum_delay_point();
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ parallel_vacuum_update_progress();
+
for (i = 0; i < nRoot; i++)
{
ginVacuumPostingTree(&gvs, rootOfPostingTree[i]);
@@ -751,6 +756,9 @@ ginvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats)
vacuum_delay_point();
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ parallel_vacuum_update_progress();
+
buffer = ReadBufferExtended(index, MAIN_FORKNUM, blkno,
RBM_NORMAL, info->strategy);
LockBuffer(buffer, GIN_SHARE);
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 0aa6e58..cb3deb1 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -17,11 +17,13 @@
#include "access/genam.h"
#include "access/gist_private.h"
#include "access/transam.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "lib/integerset.h"
#include "miscadmin.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
+#include "utils/backend_progress.h"
#include "utils/memutils.h"
/* Working state needed by gistbulkdelete */
@@ -223,7 +225,11 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
break;
/* Iterate over pages, then loop back to recheck length */
for (; blkno < num_pages; blkno++)
+ {
gistvacuumpage(&vstate, blkno, blkno);
+ if (info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ parallel_vacuum_update_progress();
+ }
}
/*
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 77fd147..d807c9f 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -505,6 +505,12 @@ loop_top:
blkno = bucket_blkno;
+ /*
+ * For hash indexes, we report parallel vacuum progress
+ * for every bucket.
+ */
+ if (info->report_parallel_progress)
+ parallel_vacuum_update_progress();
/*
* We need to acquire a cleanup lock on the primary bucket page to out
* wait concurrent scans before deleting the dead tuples.
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d59711b..f11a1b1 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -415,6 +415,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->rel = rel;
vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
&vacrel->indrels);
+
if (instrument && vacrel->nindexes > 0)
{
/* Copy index names used by instrumentation (not error reporting) */
@@ -459,6 +460,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
Assert(params->index_cleanup == VACOPTVALUE_AUTO);
}
+ /* report number of indexes to vacuum, if we are told to cleanup indexes */
+ if (vacrel->do_index_cleanup)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, vacrel->nindexes);
+
vacrel->bstrategy = bstrategy;
vacrel->relfrozenxid = rel->rd_rel->relfrozenxid;
vacrel->relminmxid = rel->rd_rel->relminmxid;
@@ -2301,6 +2306,12 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
lazy_vacuum_one_index(indrel, istat, vacrel->old_live_tuples,
vacrel);
+ /*
+ * Done vacuuming an index. Increment the indexes completed
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
+
if (lazy_check_wraparound_failsafe(vacrel))
{
/* Wraparound emergency -- end current index scan */
@@ -2335,11 +2346,14 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Assert(allindexes || vacrel->failsafe_active);
/*
- * Increase and report the number of index scans.
+ * Reset and report the number of indexes scanned.
+ * Also, increase and report the number of index
+ * scans.
*
* We deliberately include the case where we started a round of bulk
* deletes that we weren't able to finish due to the failsafe triggering.
*/
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
vacrel->num_index_scans++;
pgstat_progress_update_param(PROGRESS_VACUUM_NUM_INDEX_VACUUMS,
vacrel->num_index_scans);
@@ -2593,10 +2607,17 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
{
vacrel->failsafe_active = true;
- /* Disable index vacuuming, index cleanup, and heap rel truncation */
+ /*
+ * Disable index vacuuming, index cleanup, and heap rel truncation
+ *
+ * Also, report to progress.h that we are no longer tracking
+ * index vacuum/cleanup.
+ */
vacrel->do_index_vacuuming = false;
vacrel->do_index_cleanup = false;
vacrel->do_rel_truncate = false;
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_TOTAL, 0);
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, 0);
ereport(WARNING,
(errmsg("bypassing nonessential maintenance of table \"%s.%s.%s\" as a failsafe after %d index scans",
@@ -2644,6 +2665,12 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
vacrel->indstats[idx] =
lazy_cleanup_one_index(indrel, istat, reltuples,
estimated_count, vacrel);
+
+ /*
+ * Done cleaning an index. Increment the indexes completed
+ */
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED,
+ idx + 1);
}
}
else
@@ -2678,6 +2705,7 @@ lazy_vacuum_one_index(Relation indrel, IndexBulkDeleteResult *istat,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = true;
ivinfo.message_level = DEBUG2;
ivinfo.num_heap_tuples = reltuples;
@@ -2726,6 +2754,7 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
ivinfo.index = indrel;
ivinfo.analyze_only = false;
ivinfo.report_progress = false;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = estimated_count;
ivinfo.message_level = DEBUG2;
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index b52eca8..39349bb 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -998,6 +998,8 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
if (info->report_progress)
pgstat_progress_update_param(PROGRESS_SCAN_BLOCKS_DONE,
scanblkno);
+ if (info->report_parallel_progress && (scanblkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ parallel_vacuum_update_progress();
}
}
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index ad90b21..0589683 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -21,11 +21,13 @@
#include "access/transam.h"
#include "access/xloginsert.h"
#include "catalog/storage_xlog.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/indexfsm.h"
#include "storage/lmgr.h"
+#include "utils/backend_progress.h"
#include "utils/snapmgr.h"
@@ -843,6 +845,9 @@ spgvacuumscan(spgBulkDeleteState *bds)
/* empty the pending-list after each page */
if (bds->pendingList != NULL)
spgprocesspending(bds);
+ /* report parallel vacuum progress */
+ if (bds->info->report_parallel_progress && (blkno % REPORT_PARALLEL_VACUUM_EVERY_PAGES) == 0)
+ parallel_vacuum_update_progress();
}
}
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 61f1d39..11b3212 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -3348,6 +3348,7 @@ validate_index(Oid heapId, Oid indexId, Snapshot snapshot)
ivinfo.index = indexRelation;
ivinfo.analyze_only = false;
ivinfo.report_progress = true;
+ ivinfo.report_parallel_progress = false;
ivinfo.estimated_count = true;
ivinfo.message_level = DEBUG2;
ivinfo.num_heap_tuples = heapRelation->rd_rel->reltuples;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2d8104b..c37b20b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1165,7 +1165,8 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples,
+ S.param8 AS indexes_total, S.param9 AS indexes_completed
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index a6d5ed1..2da956f 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -75,6 +75,8 @@ int vacuum_multixact_failsafe_age;
static MemoryContext vac_context = NULL;
static BufferAccessStrategy vac_strategy;
+/* Shared parameter to track vacuum parallel progress */
+pg_atomic_uint32 *ParallelVacuumProgress = NULL;
/*
* Variables for cost-based parallel vacuum. See comments atop
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index f26d796..b5b80c9 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -30,6 +30,7 @@
#include "access/table.h"
#include "access/xact.h"
#include "catalog/index.h"
+#include "commands/progress.h"
#include "commands/vacuum.h"
#include "optimizer/paths.h"
#include "pgstat.h"
@@ -50,6 +51,8 @@
#define PARALLEL_VACUUM_KEY_WAL_USAGE 5
#define PARALLEL_VACUUM_KEY_INDEX_STATS 6
+#define PARALLEL_VACUUM_PROGRESS_TIMEOUT 1000
+
/*
* Shared information among parallel workers. So this is allocated in the DSM
* segment.
@@ -103,6 +106,17 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /*
+ * Counter for vacuuming and cleanup progress reporting.
+ * This value is used to report index vacuum/cleanup progress
+ * in parallel_vacuum_progress_report. We keep this
+ * counter to avoid having to loop through
+ * ParallelVacuumState->indstats to determine the number
+ * of indexes completed.
+ */
+ pg_atomic_uint32 idx_completed_progress;
+
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -213,6 +227,7 @@ static void parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation
static bool parallel_vacuum_index_is_parallel_safe(Relation indrel, int num_index_scans,
bool vacuum);
static void parallel_vacuum_error_callback(void *arg);
+static void parallel_wait_for_workers_to_finish(ParallelVacuumState *pvs);
/*
* Try to enter parallel mode and create a parallel context. Then initialize
@@ -364,6 +379,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
pg_atomic_init_u32(&(shared->idx), 0);
+ pg_atomic_init_u32(&(shared->idx_completed_progress), 0);
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
@@ -618,8 +634,9 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
vacuum));
}
- /* Reset the parallel index processing counter */
+ /* Reset the parallel index processing counter ( index progress counter also ) */
pg_atomic_write_u32(&(pvs->shared->idx), 0);
+ pg_atomic_write_u32(&(pvs->shared->idx_completed_progress), 0);
/* Setup the shared cost-based vacuum delay and launch workers */
if (nworkers > 0)
@@ -645,6 +662,14 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
LaunchParallelWorkers(pvs->pcxt);
+ /*
+ * Set the shared parallel vacuum progress. This will be used
+ * to periodically update progress.h with completed indexes
+ * in a parallel vacuum. See comments in parallel_vacuum_update_progress
+ * for more details.
+ */
+ ParallelVacuumProgress = &(pvs->shared->idx_completed_progress);
+
if (pvs->pcxt->nworkers_launched > 0)
{
/*
@@ -688,7 +713,21 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
*/
if (nworkers > 0)
{
- /* Wait for all vacuum workers to finish */
+ /*
+ * To wait for parallel workers to finish,
+ * first call parallel_wait_for_workers_to_finish
+ * which is responsible for reporting the
+ * number of indexes completed.
+ *
+ * Afterwards, WaitForParallelWorkersToFinish is called
+ * to do the real work of waiting for parallel workers
+ * to finish.
+ *
+ * Note: Both routines will acquire a WaitLatch in their
+ * respective loops.
+ */
+ parallel_wait_for_workers_to_finish(pvs);
+
WaitForParallelWorkersToFinish(pvs->pcxt);
for (int i = 0; i < pvs->pcxt->nworkers_launched; i++)
@@ -710,6 +749,9 @@ parallel_vacuum_process_all_indexes(ParallelVacuumState *pvs, int num_index_scan
indstats->status = PARALLEL_INDVAC_STATUS_INITIAL;
}
+ /* Reset parallel progress */
+ ParallelVacuumProgress = NULL;
+
/*
* Carry the shared balance value to heap scan and disable shared costing
*/
@@ -838,7 +880,8 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
ivinfo.estimated_count = pvs->shared->estimated_count;
ivinfo.num_heap_tuples = pvs->shared->reltuples;
ivinfo.strategy = pvs->bstrategy;
-
+ /* Only the leader should report parallel vacuum progress */
+ ivinfo.report_parallel_progress = !IsParallelWorker();
/* Update error traceback information */
pvs->indname = pstrdup(RelationGetRelationName(indrel));
pvs->status = indstats->status;
@@ -857,6 +900,9 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
RelationGetRelationName(indrel));
}
+ if (ivinfo.report_parallel_progress)
+ parallel_vacuum_update_progress();
+
/*
* Copy the index bulk-deletion result returned from ambulkdelete and
* amvacuumcleanup to the DSM segment if it's the first cycle because they
@@ -888,6 +934,9 @@ parallel_vacuum_process_one_index(ParallelVacuumState *pvs, Relation indrel,
pvs->status = PARALLEL_INDVAC_STATUS_COMPLETED;
pfree(pvs->indname);
pvs->indname = NULL;
+
+ /* Update the number of indexes completed. */
+ pg_atomic_add_fetch_u32(&(pvs->shared->idx_completed_progress), 1);
}
/*
@@ -1072,3 +1121,58 @@ parallel_vacuum_error_callback(void *arg)
return;
}
}
+
+/*
+ * Read the shared ParallelVacuumProgress and update progress.h
+ * with indexes vacuumed so far. This function is called periodically
+ * by index AMs as well as parallel_vacuum_process_one_index.
+ *
+ * To avoid unnecessarily updating progress, we check the progress
+ * values from the backend entry and only update if the value
+ * of completed indexes increases.
+ *
+ * Note: This function should be used by the leader process only,
+ * and it's up to the caller to ensure this.
+ */
+void
+parallel_vacuum_update_progress(void)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+
+ Assert(!IsParallelWorker);
+
+ if (beentry && ParallelVacuumProgress)
+ {
+ int parallel_vacuum_current_value = beentry->st_progress_param[PROGRESS_VACUUM_INDEX_COMPLETED];
+ int parallel_vacuum_new_value = pg_atomic_read_u32(ParallelVacuumProgress);
+
+ if (parallel_vacuum_new_value > parallel_vacuum_current_value)
+ pgstat_progress_update_param(PROGRESS_VACUUM_INDEX_COMPLETED, parallel_vacuum_new_value);
+ }
+}
+
+/*
+ * Check if we are done vacuuming indexes and report
+ * progress.
+ *
+ * We nap using with a WaitLatch to avoid a busy loop.
+ *
+ * Note: This function should be used by the leader process only,
+ * and it's up to the caller to ensure this.
+ */
+void
+parallel_wait_for_workers_to_finish(ParallelVacuumState *pvs)
+{
+ Assert(!IsParallelWorker);
+
+ while (pg_atomic_read_u32(ParallelVacuumProgress) < pvs->nindexes)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ parallel_vacuum_update_progress();
+
+ (void) WaitLatch(MyLatch, WL_TIMEOUT | WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, PARALLEL_VACUUM_PROGRESS_TIMEOUT,
+ WAIT_EVENT_PARALLEL_VACUUM_FINISH);
+ ResetLatch(MyLatch);
+ }
+}
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index b2abd75..eeba2be 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -460,6 +460,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_XACT_GROUP_UPDATE:
event_name = "XactGroupUpdate";
break;
+ case WAIT_EVENT_PARALLEL_VACUUM_FINISH:
+ event_name = "ParallelVacuumFinish";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index e1c4fdb..7474734 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -46,6 +46,7 @@ typedef struct IndexVacuumInfo
Relation index; /* the index being vacuumed */
bool analyze_only; /* ANALYZE (without any actual vacuum) */
bool report_progress; /* emit progress.h status reports */
+ bool report_parallel_progress; /* emit progress.h status reports for parallel vacuum */
bool estimated_count; /* num_heap_tuples is an estimate */
int message_level; /* ereport level for progress messages */
double num_heap_tuples; /* tuples remaining in heap */
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index a28938c..0e97c6d 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,8 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_INDEX_TOTAL 7
+#define PROGRESS_VACUUM_INDEX_COMPLETED 8
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 4e4bc26..5a6b454 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -64,6 +64,12 @@
/* value for checking vacuum flags */
#define VACUUM_OPTION_MAX_VALID_VALUE ((1 << 3) - 1)
+/*
+ * Parallel Index vacuum progress is reported every 1GB of blocks
+ * scanned.
+ */
+#define REPORT_PARALLEL_VACUUM_EVERY_PAGES ((BlockNumber) (((uint64) 1024 * 1024 * 1024) / BLCKSZ))
+
/* Abstract type for parallel vacuum state */
typedef struct ParallelVacuumState ParallelVacuumState;
@@ -259,6 +265,8 @@ extern PGDLLIMPORT int vacuum_multixact_freeze_table_age;
extern PGDLLIMPORT int vacuum_failsafe_age;
extern PGDLLIMPORT int vacuum_multixact_failsafe_age;
+extern PGDLLIMPORT pg_atomic_uint32 *ParallelVacuumProgress;
+
/* Variables for cost-based parallel vacuum */
extern PGDLLIMPORT pg_atomic_uint32 *VacuumSharedCostBalance;
extern PGDLLIMPORT pg_atomic_uint32 *VacuumActiveNWorkers;
@@ -333,5 +341,6 @@ extern bool std_typanalyze(VacAttrStats *stats);
extern double anl_random_fract(void);
extern double anl_init_selection_state(int n);
extern double anl_get_next_S(double t, int n, double *stateptr);
+extern void parallel_vacuum_update_progress(void);
#endif /* VACUUM_H */
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 0b2100b..95e9fef 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -128,7 +128,8 @@ typedef enum
WAIT_EVENT_SYNC_REP,
WAIT_EVENT_WAL_RECEIVER_EXIT,
WAIT_EVENT_WAL_RECEIVER_WAIT_START,
- WAIT_EVENT_XACT_GROUP_UPDATE
+ WAIT_EVENT_XACT_GROUP_UPDATE,
+ WAIT_EVENT_PARALLEL_VACUUM_FINISH
} WaitEventIPC;
/* ----------
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 37c1c86..896043e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2021,7 +2021,9 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param7 AS num_dead_tuples,
+ s.param8 AS indexes_total,
+ s.param9 AS indexes_completed
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT s.stats_reset,
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-04-06 21:22 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-14 01:32 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Robert Haas <[email protected]>
2022-06-20 06:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-11 13:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-11-08 07:49 ` Masahiko Sawada <[email protected]>
2 siblings, 0 replies; 74+ messages in thread
From: Masahiko Sawada @ 2022-11-08 07:49 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; Peter Geoghegan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On Thu, Nov 3, 2022 at 1:52 AM Imseih (AWS), Sami <[email protected]> wrote:
>
> Attached is v13-0001--Show-progress-for-index-vacuums.patch which addresses
> the latest comments.
Thank you for updating the patch!
> 4/ Went back to calling parallel_vacuum_progress_report inside
> WaitForParallelWorkersToFinish to cover the case when a
> leader is waiting for parallel workers to finish.
I don't think we need to modify WaitForParallelWorkersToFinish to
cover that case. Instead, I think the leader process can execute a new
function. The function will be like WaitForParallelWorkersToFinish()
but simpler; it just updates the progress information if necessary and
then checks if idx_completed_progress is equal to the number of
indexes to vacuum. If yes, return from the function and call
WaitForParallelWorkersToFinish() to wait for all workers to finish.
Otherwise, it naps by using WaitLatch() and does this loop again.
---
@@ -46,6 +46,8 @@ typedef struct ParallelContext
ParallelWorkerInfo *worker;
int nknown_attached_workers;
bool *known_attached_workers;
+ void (*parallel_progress_callback)(void *arg);
+ void *parallel_progress_arg;
} ParallelContext;
With the above change I suggested, I think we won't need to have a
callback function in ParallelContext. Instead, I think we can have
index-AMs call parallel_vacuum_report() if report_parallel_progress is
true.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
@ 2022-04-05 17:31 ` Andres Freund <[email protected]>
2022-04-05 18:45 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
1 sibling, 1 reply; 74+ messages in thread
From: Andres Freund @ 2022-04-05 17:31 UTC (permalink / raw)
To: Imseih (AWS), Sami <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>
On 2022-04-05 16:42:28 +0000, Imseih (AWS), Sami wrote:
> > Why isn't the obvious thing to do here to provide a way to associate workers
> > with their leaders in shared memory, but to use the existing progress fields
> > to report progress? Then, when querying progress, the leader and workers
> > progress fields can be combined to show the overall progress?
>
> The original intent was this, however the workers
> can exit before the command completes and the
> worker progress data will be lost.
Can't the progress data trivially be inferred by the fact that the worker
completed?
^ permalink raw reply [nested|flat] 74+ messages in thread
* Re: Add index scan progress to pg_stat_progress_vacuum
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-12 07:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Re: Add index scan progress to pg_stat_progress_vacuum Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-03 05:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Re: Add index scan progress to pg_stat_progress_vacuum Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-14 01:44 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Re: Add index scan progress to pg_stat_progress_vacuum Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
2022-04-05 16:42 ` Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-04-05 17:31 ` Re: Add index scan progress to pg_stat_progress_vacuum Andres Freund <[email protected]>
@ 2022-04-05 18:45 ` Imseih (AWS), Sami <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Imseih (AWS), Sami @ 2022-04-05 18:45 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Geoghegan <[email protected]>; Bossart, Nathan <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>
> Can't the progress data trivially be inferred by the fact that the worker
> completed?
Yes, at some point, this idea was experimented with in
0004-Expose-progress-for-the-vacuuming-indexes-cleanup-ph.patch.
This patch did the calculation in system_views.sql
However, the view is complex and there could be some edge
cases with inferring the values that lead to wrong values being reported.
Regards,
Sami Imseih
Amazon Web Services
^ permalink raw reply [nested|flat] 74+ messages in thread
* [PATCH] Provide a pg_truncate_freespacemap function.
@ 2024-03-01 07:57 Ronan Dunklau <[email protected]>
0 siblings, 0 replies; 74+ messages in thread
From: Ronan Dunklau @ 2024-03-01 07:57 UTC (permalink / raw)
Similar to the pg_truncate_visibilitymap function, this one allows to
avoid downtime to fix an FSM in the case a corruption event happens
---
contrib/pg_freespacemap/Makefile | 5 +-
.../pg_freespacemap--1.2--1.3.sql | 12 ++++
contrib/pg_freespacemap/pg_freespacemap.c | 64 +++++++++++++++++++
.../pg_freespacemap/pg_freespacemap.control | 2 +-
4 files changed, 80 insertions(+), 3 deletions(-)
create mode 100644 contrib/pg_freespacemap/pg_freespacemap--1.2--1.3.sql
diff --git a/contrib/pg_freespacemap/Makefile b/contrib/pg_freespacemap/Makefile
index b48e4b255bc..0f4b52f5812 100644
--- a/contrib/pg_freespacemap/Makefile
+++ b/contrib/pg_freespacemap/Makefile
@@ -6,8 +6,9 @@ OBJS = \
pg_freespacemap.o
EXTENSION = pg_freespacemap
-DATA = pg_freespacemap--1.1.sql pg_freespacemap--1.1--1.2.sql \
- pg_freespacemap--1.0--1.1.sql
+DATA = pg_freespacemap--1.1.sql pg_freespacemap--1.2--1.3.sql \
+ pg_freespacemap--1.1--1.2.sql \
+ pg_freespacemap--1.0--1.1.sql
PGFILEDESC = "pg_freespacemap - monitoring of free space map"
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_freespacemap/pg_freespacemap.conf
diff --git a/contrib/pg_freespacemap/pg_freespacemap--1.2--1.3.sql b/contrib/pg_freespacemap/pg_freespacemap--1.2--1.3.sql
new file mode 100644
index 00000000000..cc94eccf2f6
--- /dev/null
+++ b/contrib/pg_freespacemap/pg_freespacemap--1.2--1.3.sql
@@ -0,0 +1,12 @@
+/* contrib/pg_freespacemap/pg_freespacemap--1.2--1.3.sql */
+
+-- complain if script is sourced in psql, rather than via ALTER EXTENSION
+\echo Use "ALTER EXTENSION pg_freespacemap UPDATE TO '1.3'" to load this file. \quit
+
+CREATE FUNCTION pg_truncate_freespace_map(regclass)
+RETURNS void
+AS 'MODULE_PATHNAME', 'pg_truncate_freespace_map'
+LANGUAGE C STRICT
+PARALLEL UNSAFE;
+
+REVOKE ALL ON FUNCTION pg_truncate_freespace_map(regclass) FROM PUBLIC;
diff --git a/contrib/pg_freespacemap/pg_freespacemap.c b/contrib/pg_freespacemap/pg_freespacemap.c
index b82cab2d97e..fa492c18534 100644
--- a/contrib/pg_freespacemap/pg_freespacemap.c
+++ b/contrib/pg_freespacemap/pg_freespacemap.c
@@ -9,8 +9,12 @@
#include "postgres.h"
#include "access/relation.h"
+#include "access/xloginsert.h"
+#include "catalog/storage_xlog.h"
#include "funcapi.h"
#include "storage/freespace.h"
+#include "storage/smgr.h"
+#include "utils/rel.h"
PG_MODULE_MAGIC;
@@ -20,6 +24,8 @@ PG_MODULE_MAGIC;
*/
PG_FUNCTION_INFO_V1(pg_freespace);
+PG_FUNCTION_INFO_V1(pg_truncate_freespace_map);
+
Datum
pg_freespace(PG_FUNCTION_ARGS)
{
@@ -40,3 +46,61 @@ pg_freespace(PG_FUNCTION_ARGS)
relation_close(rel, AccessShareLock);
PG_RETURN_INT16(freespace);
}
+
+
+/*
+ * Remove the free space map for a relation.
+ *
+ * This is useful in case of corruption of the FSM, as the
+ * only other options are either triggering a VACUUM FULL or
+ * shutting down the server and removing the FSM on the filesystem.
+ */
+Datum
+pg_truncate_freespace_map(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ Relation rel;
+ ForkNumber fork;
+ BlockNumber block;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /* Only some relkinds have a freespace map */
+ if (!RELKIND_HAS_TABLE_AM(rel->rd_rel->relkind))
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("relation \"%s\" is of wrong relation kind",
+ RelationGetRelationName(rel)),
+ errdetail_relkind_not_supported(rel->rd_rel->relkind)));
+
+
+ /* Forcibly reset cached file size */
+ RelationGetSmgr(rel)->smgr_cached_nblocks[FSM_FORKNUM] = InvalidBlockNumber;
+
+ /* Just pretend we're going to wipeout the whole rel */
+ block = FreeSpaceMapPrepareTruncateRel(rel, 0);
+
+ if (BlockNumberIsValid(block))
+ {
+ fork = FSM_FORKNUM;
+ smgrtruncate(RelationGetSmgr(rel), &fork, 1, &block);
+ }
+
+ if (RelationNeedsWAL(rel))
+ {
+ xl_smgr_truncate xlrec;
+
+ xlrec.blkno = 0;
+ xlrec.rlocator = rel->rd_locator;
+ xlrec.flags = SMGR_TRUNCATE_FSM;
+
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, sizeof(xlrec));
+
+ XLogInsert(RM_SMGR_ID, XLOG_SMGR_TRUNCATE | XLR_SPECIAL_REL_UPDATE);
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/contrib/pg_freespacemap/pg_freespacemap.control b/contrib/pg_freespacemap/pg_freespacemap.control
index ac8fc5050a9..1992320691b 100644
--- a/contrib/pg_freespacemap/pg_freespacemap.control
+++ b/contrib/pg_freespacemap/pg_freespacemap.control
@@ -1,5 +1,5 @@
# pg_freespacemap extension
comment = 'examine the free space map (FSM)'
-default_version = '1.2'
+default_version = '1.3'
module_pathname = '$libdir/pg_freespacemap'
relocatable = true
--
2.44.0
--nextPart2183423.Mh6RI2rZIc--
^ permalink raw reply [nested|flat] 74+ messages in thread
end of thread, other threads:[~2024-03-01 07:57 UTC | newest]
Thread overview: 74+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-01-11 01:01 Re: Add index scan progress to pg_stat_progress_vacuum Imseih (AWS), Sami <[email protected]>
2022-01-11 19:01 ` Bossart, Nathan <[email protected]>
2022-01-11 20:33 ` Imseih (AWS), Sami <[email protected]>
2022-01-11 22:18 ` Bossart, Nathan <[email protected]>
2022-01-12 07:44 ` Masahiko Sawada <[email protected]>
2022-01-12 19:28 ` Bossart, Nathan <[email protected]>
2022-01-13 03:52 ` Imseih (AWS), Sami <[email protected]>
2022-01-27 02:07 ` Imseih (AWS), Sami <[email protected]>
2022-01-27 02:15 ` Imseih (AWS), Sami <[email protected]>
2022-01-27 21:09 ` Imseih (AWS), Sami <[email protected]>
2022-02-01 20:33 ` Imseih (AWS), Sami <[email protected]>
2022-02-10 19:39 ` Imseih (AWS), Sami <[email protected]>
2022-02-17 13:52 ` Imseih (AWS), Sami <[email protected]>
2022-02-21 19:03 ` Imseih (AWS), Sami <[email protected]>
2022-02-21 19:09 ` Nathan Bossart <[email protected]>
2022-02-23 18:02 ` Imseih (AWS), Sami <[email protected]>
2022-02-23 18:41 ` Peter Geoghegan <[email protected]>
2022-02-25 19:52 ` Nathan Bossart <[email protected]>
2022-02-27 17:16 ` Imseih (AWS), Sami <[email protected]>
2022-03-03 05:08 ` Imseih (AWS), Sami <[email protected]>
2022-03-08 06:04 ` Masahiko Sawada <[email protected]>
2022-03-08 15:41 ` Imseih (AWS), Sami <[email protected]>
2022-03-09 01:53 ` Masahiko Sawada <[email protected]>
2022-03-09 02:35 ` Imseih (AWS), Sami <[email protected]>
2022-03-09 02:57 ` Imseih (AWS), Sami <[email protected]>
2022-03-09 03:15 ` Masahiko Sawada <[email protected]>
2022-03-09 22:52 ` Imseih (AWS), Sami <[email protected]>
2022-03-10 00:52 ` Nathan Bossart <[email protected]>
2022-03-10 01:22 ` Imseih (AWS), Sami <[email protected]>
2022-03-10 01:37 ` Imseih (AWS), Sami <[email protected]>
2022-03-10 21:30 ` Imseih (AWS), Sami <[email protected]>
2022-03-10 22:36 ` Nathan Bossart <[email protected]>
2022-03-12 07:00 ` Imseih (AWS), Sami <[email protected]>
2022-03-12 21:20 ` Nathan Bossart <[email protected]>
2022-03-14 01:44 ` Masahiko Sawada <[email protected]>
2022-03-14 16:20 ` Imseih (AWS), Sami <[email protected]>
2022-03-22 03:19 ` Masahiko Sawada <[email protected]>
2022-03-22 07:27 ` Imseih (AWS), Sami <[email protected]>
2022-03-22 07:48 ` Masahiko Sawada <[email protected]>
2022-03-22 21:57 ` Imseih (AWS), Sami <[email protected]>
2022-03-25 14:54 ` Masahiko Sawada <[email protected]>
2022-03-29 12:08 ` Imseih (AWS), Sami <[email protected]>
2022-03-29 12:25 ` Imseih (AWS), Sami <[email protected]>
2022-04-03 16:50 ` Andres Freund <[email protected]>
2022-04-05 16:42 ` Imseih (AWS), Sami <[email protected]>
2022-04-05 17:19 ` Robert Haas <[email protected]>
2022-04-06 21:22 ` Imseih (AWS), Sami <[email protected]>
2022-04-07 13:20 ` Robert Haas <[email protected]>
2022-04-07 15:38 ` Masahiko Sawada <[email protected]>
2022-04-07 23:25 ` Greg Stark <[email protected]>
2022-04-14 01:32 ` Imseih (AWS), Sami <[email protected]>
2022-05-02 03:30 ` Masahiko Sawada <[email protected]>
2022-05-05 19:26 ` Imseih (AWS), Sami <[email protected]>
2022-05-26 13:41 ` Imseih (AWS), Sami <[email protected]>
2022-05-26 15:43 ` Masahiko Sawada <[email protected]>
2022-05-27 01:52 ` Imseih (AWS), Sami <[email protected]>
2022-06-03 05:39 ` Masahiko Sawada <[email protected]>
2022-06-06 14:41 ` Robert Haas <[email protected]>
2022-06-20 06:35 ` Masahiko Sawada <[email protected]>
2022-08-02 18:06 ` Jacob Champion <[email protected]>
2022-10-11 13:50 ` Imseih (AWS), Sami <[email protected]>
2022-10-12 07:15 ` Masahiko Sawada <[email protected]>
2022-10-14 20:05 ` Imseih (AWS), Sami <[email protected]>
2022-11-02 16:52 ` Imseih (AWS), Sami <[email protected]>
2022-11-03 08:16 ` Ian Lawrence Barwick <[email protected]>
2022-11-04 13:27 ` Imseih (AWS), Sami <[email protected]>
2022-11-09 03:00 ` Andres Freund <[email protected]>
2022-11-11 19:10 ` Imseih (AWS), Sami <[email protected]>
2022-11-18 13:07 ` Masahiko Sawada <[email protected]>
2022-11-28 23:57 ` Imseih (AWS), Sami <[email protected]>
2022-11-08 07:49 ` Masahiko Sawada <[email protected]>
2022-04-05 17:31 ` Andres Freund <[email protected]>
2022-04-05 18:45 ` Imseih (AWS), Sami <[email protected]>
2024-03-01 07:57 [PATCH] Provide a pg_truncate_freespacemap function. Ronan Dunklau <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox